diff --git a/INSTALL b/INSTALL new file mode 100644 index 0000000000000000000000000000000000000000..651a044b10020575960ae54910d45e919a9a0930 --- /dev/null +++ b/INSTALL @@ -0,0 +1,3 @@ + + See README file. + diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..94a045322262546cfb9d72561e1d587b5c2ffb1e --- /dev/null +++ b/LICENSE @@ -0,0 +1,621 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS diff --git a/MANIFEST b/MANIFEST new file mode 100644 index 0000000000000000000000000000000000000000..b8dd1921c18c96d143f477fd52e7e9b03217d746 --- /dev/null +++ b/MANIFEST @@ -0,0 +1,13 @@ +changelog +INSTALL +README +LICENSE +MANIFEST +Makefile.PL +lib/Ora2Pg.pm +lib/Ora2Pg/PLSQL.pm +lib/Ora2Pg/GEOM.pm +lib/Ora2Pg/MySQL.pm +scripts/ora2pg +scripts/ora2pg_scanner +doc/Ora2Pg.pod diff --git a/Makefile.PL b/Makefile.PL new file mode 100644 index 0000000000000000000000000000000000000000..72d8b2e7926b4b8842eb3ad344fed878ed5c555c --- /dev/null +++ b/Makefile.PL @@ -0,0 +1,1407 @@ +use ExtUtils::MakeMaker qw(prompt WriteMakefile); + +my @ALLOWED_ARGS = ('CONFDIR','DOCDIR','DESTDIR','QUIET','INSTALLDIRS','INSTALL_BASE','PREFIX'); + +# Parse command line arguments and store them as environment variables +while ($_ = shift) { + my ($k,$v) = split(/=/, $_, 2); + if (grep(/^$k$/, @ALLOWED_ARGS)) { + $ENV{$k} = $v; + } +} + +# Default install path +my $CONFDIR = $ENV{CONFDIR} || '/etc/ora2pg'; +my $RPM_CONFDIR = $CONFDIR; +my $DOCDIR = $ENV{DOCDIR} || '/usr/local/share/doc/ora2pg'; +my $DEST_CONF_FILE = 'ora2pg.conf.dist'; +my $DATA_LIMIT_DEFAULT = 10000; +if ($^O =~ /MSWin32|dos/i) { + $DEST_CONF_FILE = 'ora2pg_dist.conf'; + $DATA_LIMIT_DEFAULT = 2000; +} + +my $PREFIX = $ENV{DESTDIR} || $ENV{PREFIX} || $ENV{INSTALL_BASE} || ''; +$PREFIX =~ s/\/$//; +$ENV{INSTALLDIRS} ||= 'site'; + +# Try to set the default configuration directory following $PREFIX +if ($^O =~ /MSWin32|dos/i) { + # Force default path + $CONFDIR = 'C:\ora2pg'; + $DOCDIR = 'C:\ora2pg'; +} elsif ($PREFIX) { + if (!$ENV{CONFDIR}) { + $CONFDIR = $PREFIX . '/etc/ora2pg'; + } else { + $CONFDIR = $PREFIX . '/' . $ENV{CONFDIR}; + } + if (!$ENV{DOCDIR}) { + $DOCDIR = $PREFIX . '/doc/ora2pg'; + } else { + $DOCDIR = $PREFIX . '/' . $ENV{DOCDIR}; + } +} + +# Try to find all binary used by Ora2Pg +my $bzip2 = ''; +if ($^O !~ /MSWin32|dos/i) { + my $bzip2 = `which bzip2`; + chomp($bzip2); + $bzip2 ||= '/usr/bin/bzip2'; +} + +my $oracle_home = $ENV{ORACLE_HOME} || '/usr/local/oracle/10g'; + +# Setup ok. generating default ora2pg.conf config file +unless(open(OUTCFG, ">$DEST_CONF_FILE")) { + print "\nError: can't write config file $DEST_CONF_FILE, $!\n"; + exit 0; +} + + print OUTCFG qq{ +#################### Ora2Pg Configuration file ##################### + +# Support for including a common config file that may contain any +# of the following configuration directives. +#IMPORT common.conf + +#------------------------------------------------------------------------------ +# INPUT SECTION (Oracle connection or input file) +#------------------------------------------------------------------------------ + +# Set this directive to a file containing PL/SQL Oracle Code like function, +# procedure or a full package body to prevent Ora2Pg from connecting to an +# Oracle database end just apply his conversion tool to the content of the +# file. This can only be used with the following export type: PROCEDURE, +# FUNCTION or PACKAGE. If you don't know what you do don't use this directive. +#INPUT_FILE ora_plsql_src.sql + +# Set the Oracle home directory +ORACLE_HOME $oracle_home + +# Set Oracle database connection (datasource, user, password) +ORACLE_DSN dbi:Oracle:host=mydb.mydom.fr;sid=SIDNAME;port=1521 +ORACLE_USER system +ORACLE_PWD manager + +# Set this to 1 if you connect as simple user and can not extract things +# from the DBA_... tables. It will use tables ALL_... This will not works +# with GRANT export, you should use an Oracle DBA username at ORACLE_USER +USER_GRANTS 0 + +# Trace all to stderr +DEBUG 0 + +# This directive can be used to send an initial command to Oracle, just after +# the connection. For example to unlock a policy before reading objects or +# to set some session parameters. This directive can be used multiple time. +#ORA_INITIAL_COMMAND + + +#------------------------------------------------------------------------------ +# SCHEMA SECTION (Oracle schema to export and use of schema in PostgreSQL) +#------------------------------------------------------------------------------ + +# Export Oracle schema to PostgreSQL schema +EXPORT_SCHEMA 0 + +# Oracle schema/owner to use +#SCHEMA SCHEMA_NAME + +# Enable/disable the CREATE SCHEMA SQL order at starting of the output file. +# It is enable by default and concern on TABLE export type. +CREATE_SCHEMA 1 + +# Enable this directive to force Oracle to compile schema before exporting code. +# When this directive is enabled and SCHEMA is set to a specific schema name, +# only invalid objects in this schema will be recompiled. If SCHEMA is not set +# then all schema will be recompiled. To force recompile invalid object in a +# specific schema, set COMPILE_SCHEMA to the schema name you want to recompile. +# This will ask to Oracle to validate the PL/SQL that could have been invalidate +# after a export/import for example. The 'VALID' or 'INVALID' status applies to +# functions, procedures, packages and user defined types. +COMPILE_SCHEMA 1 + +# By default if you set EXPORT_SCHEMA to 1 the PostgreSQL search_path will be +# set to the schema name exported set as value of the SCHEMA directive. You can +# defined/force the PostgreSQL schema to use by using this directive. +# +# The value can be a comma delimited list of schema but not when using TABLE +# export type because in this case it will generate the CREATE SCHEMA statement +# and it doesn't support multiple schema name. For example, if you set PG_SCHEMA +# to something like "user_schema, public", the search path will be set like this +# SET search_path = user_schema, public; +# forcing the use of an other schema (here user_schema) than the one from Oracle +# schema set in the SCHEMA directive. You can also set the default search_path +# for the PostgreSQL user you are using to connect to the destination database +# by using: +# ALTER ROLE username SET search_path TO user_schema, public; +#in this case you don't have to set PG_SCHEMA. +#PG_SCHEMA + +# Use this directive to add a specific schema to the search path to look +# for PostGis functions. +#POSTGIS_SCHEMA + +# Allow to add a comma separated list of system user to exclude from +# Oracle extraction. Oracle have many of them following the modules +# installed. By default it will suppress all object owned by the following +# system users: +# 'SYSTEM','CTXSYS','DBSNMP','EXFSYS','LBACSYS','MDSYS','MGMT_VIEW', +# 'OLAPSYS','ORDDATA','OWBSYS','ORDPLUGINS','ORDSYS','OUTLN', +# 'SI_INFORMTN_SCHEMA','SYS','SYSMAN','WK_TEST','WKSYS','WKPROXY', +# 'WMSYS','XDB','APEX_PUBLIC_USER','DIP','FLOWS_020100','FLOWS_030000', +# 'FLOWS_040100','FLOWS_010600','FLOWS_FILES','MDDATA','ORACLE_OCM', +# 'SPATIAL_CSW_ADMIN_USR','SPATIAL_WFS_ADMIN_USR','XS\$NULL','PERFSTAT', +# 'SQLTXPLAIN','DMSYS','TSMSYS','WKSYS','APEX_040000','APEX_040200', +# 'DVSYS','OJVMSYS','GSMADMIN_INTERNAL','APPQOSSYS','DVSYS','DVF', +# 'AUDSYS','APEX_030200','MGMT_VIEW','ODM','ODM_MTR','TRACESRV','MTMSYS', +# 'OWBSYS_AUDIT','WEBSYS','WK_PROXY','OSE\$HTTP\$ADMIN', +# 'AURORA\$JIS\$UTILITY\$','AURORA\$ORB\$UNAUTHENTICATED', +# 'DBMS_PRIVILEGE_CAPTURE','CSMIG','MGDSYS','SDE','DBSFWUSER' +# Other list of users set to this directive will be added to this list. +#SYSUSERS OE,HR + + +# List of schema to get functions/procedures meta information that are used +# in the current schema export. When replacing call to function with OUT +# parameters, if a function is declared in an other package then the function +# call rewriting can not be done because Ora2Pg only know about functions +# declared in the current schema. By setting a comma separated list of schema +# as value of this directive, Ora2Pg will look forward in these packages for +# all functions/procedures/packages declaration before proceeding to current +# schema export. +#LOOK_FORWARD_FUNCTION SCOTT,OE + +# Force Ora2Pg to not look for function declaration. Note that this will prevent +# Ora2Pg to rewrite function replacement call if needed. Do not enable it unless +# looking forward at function breaks other export. +NO_FUNCTION_METADATA 0 + +#------------------------------------------------------------------------------ +# ENCODING SECTION (Define client encoding at Oracle and PostgreSQL side) +#------------------------------------------------------------------------------ + +# Enforce default language setting following the Oracle database encoding. This +# may be used with multibyte characters like UTF8. Here are the default values +# used by Ora2Pg, you may not change them unless you have problem with this +# encoding. This will set \$ENV{NLS_LANG} to the given value. +#NLS_LANG AMERICAN_AMERICA.AL32UTF8 +# This will set \$ENV{NLS_NCHAR} to the given value. +#NLS_NCHAR AL32UTF8 + +# By default PostgreSQL client encoding is automatically set to UTF8 to avoid +# encoding issue. If you have changed the value of NLS_LANG you might have to +# change the encoding of the PostgreSQL client. +#CLIENT_ENCODING UTF8 + + +#------------------------------------------------------------------------------ +# EXPORT SECTION (Export type and filters) +#------------------------------------------------------------------------------ + +# Type of export. Values can be the following keyword: +# TABLE Export tables, constraints, indexes, ... +# PACKAGE Export packages +# INSERT Export data from table as INSERT statement +# COPY Export data from table as COPY statement +# VIEW Export views +# GRANT Export grants +# SEQUENCE Export sequences +# TRIGGER Export triggers +# FUNCTION Export functions +# PROCEDURE Export procedures +# TABLESPACE Export tablespace (PostgreSQL >= 8 only) +# TYPE Export user defined Oracle types +# PARTITION Export range or list partition (PostgreSQL >= v8.4) +# FDW Export table as foreign data wrapper tables +# MVIEW Export materialized view as snapshot refresh view +# QUERY Convert Oracle SQL queries from a file. +# KETTLE Generate XML ktr template files to be used by Kettle. +# DBLINK Generate oracle foreign data wrapper server to use as dblink. +# SYNONYM Export Oracle's synonyms as views on other schema's objects. +# DIRECTORY Export Oracle's directories as external_file extension objects. +# LOAD Dispatch a list of queries over multiple PostgreSQl connections. +# TEST perform a diff between Oracle and PostgreSQL database. +# TEST_VIEW perform a count on both side of rows returned by views + +TYPE TABLE + +# Set this to 1 if you don't want to export comments associated to tables and +# column definitions. Default is enabled. +DISABLE_COMMENT 0 + +# Set which object to export from. By default Ora2Pg export all objects. +# Value must be a list of object name or regex separated by space. Note +# that regex will not works with 8i database, use % placeholder instead +# Ora2Pg will use the LIKE operator. There is also some extended use of +# this directive, see chapter "Limiting object to export" in documentation. +#ALLOW TABLE_TEST + +# Set which object to exclude from export process. By default none. Value +# must be a list of object name or regexp separated by space. Note that regex +# will not works with 8i database, use % placeholder instead Ora2Pg will use +# the NOT LIKE operator. There is also some extended use of this directive, +# see chapter "Limiting object to export" in documentation. +#EXCLUDE OTHER_TABLES + +# Set which view to export as table. By default none. Value must be a list of +# view name or regexp separated by space. If the object name is a view and the +# export type is TABLE, the view will be exported as a create table statement. +# If export type is COPY or INSERT, the corresponding data will be exported. +#VIEW_AS_TABLE VIEW_NAME + +# By default Ora2Pg try to order views to avoid error at import time with +# nested views. With a huge number of view this can take a very long time, +# you can bypass this ordering by enabling this directive. +NO_VIEW_ORDERING 0 + +# When exporting GRANTs you can specify a comma separated list of objects +# for which privilege will be exported. Default is export for all objects. +# Here are the possibles values TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, +# PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, DIRECTORY. Only one object +# type is allowed at a time. For example set it to TABLE if you just want to +# export privilege on tables. You can use the -g option to overwrite it. +# When used this directive prevent the export of users unless it is set to +# USER. In this case only users definitions are exported. +#GRANT_OBJECT TABLE + +# By default Ora2Pg will export your external table as file_fdw tables. If +# you don't want to export those tables at all, set the directive to 0. +EXTERNAL_TO_FDW 1 + +# Add a TRUNCATE TABLE instruction before loading data on COPY and INSERT +# export. When activated, the instruction will be added only if there's no +# global DELETE clause or one specific to the current table (see bellow). +TRUNCATE_TABLE 0 + +# Support for include a DELETE FROM ... WHERE clause filter before importing +# data and perform a delete of some lines instead of truncatinf tables. +# Value is construct as follow: TABLE_NAME[DELETE_WHERE_CLAUSE], or +# if you have only one where clause for all tables just put the delete +# clause as single value. Both are possible too. Here are some examples: +#DELETE 1=1 # Apply to all tables and delete all tuples +#DELETE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST +#DELETE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] +# The last applies two different delete where clause on tables TABLE_TEST and +# TABLE_INFO and a generic delete where clause on DATE_CREATE to all other tables. +# If TRUNCATE_TABLE is enabled it will be applied to all tables not covered by +# the DELETE definition. + +# When enabled this directive forces ora2pg to export all tables, index +# constraints, and indexes using the tablespace name defined in Oracle database. +# This works only with tablespaces that are not TEMP, USERS and SYSTEM. +USE_TABLESPACE 0 + +# Enable this directive to reorder columns and minimized the footprint +# on disk, so that more rows fit on a data page, which is the most important +# factor for speed. Default is same order than in Oracle table definition, +# that should be enough for most usage. +REORDERING_COLUMNS 0 + +# Support for include a WHERE clause filter when dumping the contents +# of tables. Value is construct as follow: TABLE_NAME[WHERE_CLAUSE], or +# if you have only one where clause for each table just put the where +# clause as value. Both are possible too. Here are some examples: +#WHERE 1=1 # Apply to all tables +#WHERE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST +#WHERE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] +# The last applies two different where clause on tables TABLE_TEST and +# TABLE_INFO and a generic where clause on DATE_CREATE to all other tables + +# Sometime you may want to extract data from an Oracle table but you need a +# a custom query for that. Not just a "SELECT * FROM table" like Ora2Pg does +# but a more complex query. This directive allows you to override the query +# used by Ora2Pg to extract data. The format is TABLENAME[SQL_QUERY]. +# If you have multiple tables to extract by replacing the Ora2Pg query, you can +# define multiple REPLACE_QUERY lines. +#REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] + +#------------------------------------------------------------------------------ +# FULL TEXT SEARCH SECTION (Control full text search export behaviors) +#------------------------------------------------------------------------------ + +# Force Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using +# pg_trgm extension. Default is to translate CONTEXT indexes into FTS indexes +# and CTXCAT indexes using pg_trgm. Most of the time using pg_trgm is enough, +# this is why this directive stand for. +# +CONTEXT_AS_TRGM 0 + +# By default Ora2Pg creates a function-based index to translate Oracle Text +# indexes. +# CREATE INDEX ON t_document +# USING gin(to_tsvector('french', title)); +# You will have to rewrite the CONTAIN() clause using to_tsvector(), example: +# SELECT id,title FROM t_document +# WHERE to_tsvector(title)) @@ to_tsquery('search_word'); +# +# To force Ora2Pg to create an extra tsvector column with a dedicated triggers +# for FTS indexes, disable this directive. In this case, Ora2Pg will add the +# column as follow: ALTER TABLE t_document ADD COLUMN tsv_title tsvector; +# Then update the column to compute FTS vectors if data have been loaded before +# UPDATE t_document SET tsv_title = +# to_tsvector('french', coalesce(title,'')); +# To automatically update the column when a modification in the title column +# appears, Ora2Pg adds the following trigger: +# +# CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS \$\$ +# BEGIN +# IF TG_OP = 'INSERT' OR new.title != old.title THEN +# new.tsv_title := +# to_tsvector('french', coalesce(new.title,'')); +# END IF; +# return new; +# END +# \$\$ LANGUAGE plpgsql; +# CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE +# ON t_document +# FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); +# +# When the Oracle text index is defined over multiple column, Ora2Pg will use +# setweight() to set a weight in the order of the column declaration. +# +FTS_INDEX_ONLY 1 + +# Use this directive to force text search configuration to use. When it is not +# set, Ora2Pg will autodetect the stemmer used by Oracle for each index and +# pg_catalog.english if nothing is found. +# +#FTS_CONFIG pg_catalog.french + +# If you want to perform your text search in an accent insensitive way, enable +# this directive. Ora2Pg will create an helper function over unaccent() and +# creates the pg_trgm indexes using this function. With FTS Ora2Pg will +# redefine your text search configuration, for example: +# +# CREATE TEXT SEARCH CONFIGURATION fr (COPY = pg_catalog.french); +# ALTER TEXT SEARCH CONFIGURATION fr +# ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; +# +# When enabled, Ora2pg will create the wrapper function: +# +# CREATE OR REPLACE FUNCTION unaccent_immutable(text) +# RETURNS text AS +# \$\$ +# SELECT public.unaccent('public.unaccent', $1) +# \$\$ LANGUAGE sql IMMUTABLE +# COST 1; +# +# indexes are exported as follow: +# +# CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document +# USING gin (unaccent_immutable(title) gin_trgm_ops); +# +# In your queries you will need to use the same function in the search to +# be able to use the function-based index. Example: +# +# SELECT * FROM t_document +# WHERE unaccent_immutable(title) LIKE '%donnees%'; +# +USE_UNACCENT 0 + +# Same as above but call lower() in the unaccent_immutable() function: +# +# CREATE OR REPLACE FUNCTION unaccent_immutable(text) +# RETURNS text AS +# \$\$ +# SELECT lower(public.unaccent('public.unaccent', $1)); +# \$\$ LANGUAGE sql IMMUTABLE; +# +USE_LOWER_UNACCENT 0 + + +#------------------------------------------------------------------------------ +# DATA DIFF SECTION (only delete and insert actually changed rows) +#------------------------------------------------------------------------------ + +# EXPERIMENTAL! Not yet working correctly with partitioned tables, parallelism, +# and direct Postgres connection! Test before using in production! +# This feature affects SQL output for data (INSERT or COPY). +# The deletion and (re-)importing of data is redirected to temporary tables +# (with configurable suffix) and matching entries (i.e. quasi-unchanged rows) +# eliminated before actual application of the DELETE, UPDATE and INSERT. +# Optional functions can be specified that are called before or after the +# actual DELETE, UPDATE and INSERT per table, or after all tables have been +# processed. +# +# Enable DATADIFF functionality +DATADIFF 0 +# Use UPDATE where changed columns can be matched by the primary key +# (otherwise rows are DELETEd and re-INSERTed, which may interfere with +# inverse foreign keys relationships!) +DATADIFF_UPDATE_BY_PKEY 0 +# Suffix for temporary tables holding rows to be deleted and to be inserted. +# Pay attention to your tables names: +# 1) There better be no two tables with names such that name1 + suffix = name2 +# 2) length(suffix) + length(tablename) < NAMEDATALEN (usually 64) +DATADIFF_DEL_SUFFIX _del +DATADIFF_UPD_SUFFIX _upd +DATADIFF_INS_SUFFIX _ins +# Allow setting the work_mem and temp_buffers parameters +# to keep temp tables in memory and have efficient sorting, etc. +DATADIFF_WORK_MEM 256 MB +DATADIFF_TEMP_BUFFERS 512 MB + +# The following are names of functions that will be called (via SELECT) +# after the temporary tables have been reduced (by removing matching rows) +# and right before or right after the actual DELETE and INSERT are performed. +# They must take four arguments, which should ideally be of type "regclass", +# representing the real table, the "deletions", the "updates", and the +# "insertions" temp table names, respectively. They are called before +# re-activation of triggers, indexes, etc. (if configured). +#DATADIFF_BEFORE my_datadiff_handler_function +#DATADIFF_AFTER my_datadiff_handler_function + +# Another function can be called (via SELECT) right before the entire COMMIT +# (i.e., after re-activation of indexes, triggers, etc.), which will be +# passed in Postgres ARRAYs of the table names of the real tables, the +# "deletions", the "updates" and the "insertions" temp tables, respectively, +# with same array index positions belonging together. So this function should +# take four arguments of type regclass[] +#DATADIFF_AFTER_ALL my_datadiff_bunch_handler_function +# If in doubt, use schema-qualified function names here. +# The search_path will have been set to PG_SCHEMA if EXPORT_SCHEMA == 1 +# (as defined by you in those config parameters, see above), +# i.e., the "public" schema is not contained if EXPORT_SCHEMA == 1 + + +#------------------------------------------------------------------------------ +# CONSTRAINT SECTION (Control constraints export and import behaviors) +#------------------------------------------------------------------------------ + +# Support for turning off certain schema features in the postgres side +# during schema export. Values can be : fkeys, pkeys, ukeys, indexes, checks +# separated by a space character. +# fkeys : turn off foreign key constraints +# pkeys : turn off primary keys +# ukeys : turn off unique column constraints +# indexes : turn off all other index types +# checks : turn off check constraints +#SKIP fkeys pkeys ukeys indexes checks + +# By default names of the primary and unique key in the source Oracle database +# are ignored and key names are autogenerated in the target PostgreSQL database +# with the PostgreSQL internal default naming rules. If you want to preserve +# Oracle primary and unique key names set this option to 1. +# Please note if value of USE_TABLESPACE is set to 1 the value of this option is +# enforced to 1 to preserve correct primary and uniqie key allocation to tablespace. +KEEP_PKEY_NAMES 0 + +# Enable this directive if you want to add primary key definitions inside the +# create table statements. If disabled (the default) primary key definition +# will be added with an alter table statement. Enable it if you are exporting +# to GreenPlum PostgreSQL database. +PKEY_IN_CREATE 0 + +# This directive allow you to add an ON UPDATE CASCADE option to a foreign +# key when a ON DELETE CASCADE is defined or always. Oracle do not support +# this feature, you have to use trigger to operate the ON UPDATE CASCADE. +# As PostgreSQL has this feature, you can choose how to add the foreign +# key option. There is three value to this directive: never, the default +# that mean that foreign keys will be declared exactly like in Oracle. +# The second value is delete, that mean that the ON UPDATE CASCADE option +# will be added only if the ON DELETE CASCADE is already defined on the +# foreign Keys. The last value, always, will force all foreign keys to be +# defined using the update option. +FKEY_ADD_UPDATE never + +# When exporting tables, Ora2Pg normally exports constraints as they are; +# if they are non-deferrable they are exported as non-deferrable. +# However, non-deferrable constraints will probably cause problems when +# attempting to import data to PostgreSQL. The following option set to 1 +# will cause all foreign key constraints to be exported as deferrable +FKEY_DEFERRABLE 0 + +# In addition when exporting data the DEFER_FKEY option set to 1 will add +# a command to defer all foreign key constraints during data export and +# the import will be done in a single transaction. This will work only if +# foreign keys have been exported as deferrable and you are not using direct +# import to PostgreSQL (PG_DSN is not defined). Constraints will then be +# checked at the end of the transaction. This directive can also be enabled +# if you want to force all foreign keys to be created as deferrable and +# initially deferred during schema export (TABLE export type). +DEFER_FKEY 0 + +# If deferring foreign keys is not possible du to the amount of data in a +# single transaction, you've not exported foreign keys as deferrable or you +# are using direct import to PostgreSQL, you can use the DROP_FKEY directive. +# It will drop all foreign keys before all data import and recreate them at +# the end of the import. +DROP_FKEY 0 + + +#------------------------------------------------------------------------------ +# TRIGGERS AND SEQUENCES SECTION (Control triggers and sequences behaviors) +#------------------------------------------------------------------------------ + +# Disables alter of sequences on all tables in COPY or INSERT mode. +# Set to 1 if you want to disable update of sequence during data migration. +DISABLE_SEQUENCE 0 + +# Disables triggers on all tables in COPY or INSERT mode. Available modes +# are USER (user defined triggers) and ALL (includes RI system +# triggers). Default is 0 do not add SQL statement to disable trigger. +# If you want to disable triggers during data migration, set the value to +# USER if your are connected as non superuser and ALL if you are connected +# as PostgreSQL superuser. A value of 1 is equal to USER. +DISABLE_TRIGGERS 0 + + +#------------------------------------------------------------------------------ +# OBJECT MODIFICATION SECTION (Control objects structure or name modifications) +#------------------------------------------------------------------------------ + +# You may wish to just extract data from some fields, the following directives +# will help you to do that. Works only with export type INSERT or COPY +# Modify output from the following tables(fields separate by space or comma) +#MODIFY_STRUCT TABLE_TEST(dico,dossier) + +# You may wish to change table names during data extraction, especally for +# replication use. Give a list of tables separate by space as follow. +#REPLACE_TABLES ORIG_TB_NAME1:NEW_TB_NAME1 ORIG_TB_NAME2:NEW_TB_NAME2 + +# You may wish to change column names during export. Give a list of tables +# and columns separate by comma as follow. +#REPLACE_COLS TB_NAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) + +# By default all object names are converted to lower case, if you +# want to preserve Oracle object name as-is set this to 1. Not recommended +# unless you always quote all tables and columns on all your scripts. +PRESERVE_CASE 0 + +# Add the given value as suffix to index names. Useful if you have indexes +# with same name as tables. Not so common but it can help. +#INDEXES_SUFFIX _idx + +# Enable this directive to rename all indexes using tablename_columns_names. +# Could be very useful for database that have multiple time the same index name +# or that use the same name than a table, which is not allowed by PostgreSQL +# Disabled by default. +INDEXES_RENAMING 0 + +# Operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops +# support B-tree indexes on the corresponding types. The difference from the +# default operator classes is that the values are compared strictly character by +# character rather than according to the locale-specific collation rules. This +# makes these operator classes suitable for use by queries involving pattern +# matching expressions (LIKE or POSIX regular expressions) when the database +# does not use the standard "C" locale. If you enable, with value 1, this will +# force Ora2Pg to export all indexes defined on varchar2() and char() columns +# using those operators. If you set it to a value greater than 1 it will only +# change indexes on columns where the charactere limit is greater or equal than +# this value. For example, set it to 128 to create these kind of indexes on +# columns of type varchar2(N) where N >= 128. +USE_INDEX_OPCLASS 0 + +# Enable this directive if you want that your partition table name will be +# exported using the parent table name. Disabled by default. If you have +# multiple partitioned table, when exported to PostgreSQL some partitions +# could have the same name but different parent tables. This is not allowed, +# table name must be unique. +PREFIX_PARTITION 0 + +# Disable this directive if your subpartitions are dedicated to your partition +# (in case of your partition_name is a part of your subpartition_name) +PREFIX_SUB_PARTITION 1 + +# If you don't want to reproduce the partitioning like in Oracle and want to +# export all partitionned Oracle data into the main single table in PostgreSQL +# enable this directive. Ora2Pg will export all data into the main table name. +# Default is to use partitionning, Ora2Pg will export data from each partition +# and import them into the PostgreSQL dedicated partition table. +DISABLE_PARTITION 0 + +# Activating this directive will force Ora2Pg to add WITH (OIDS) when creating +# tables or views as tables. Default is same as PostgreSQL, disabled. +WITH_OID 0 + +# Allow escaping of column name using Oracle reserved words. +ORA_RESERVED_WORDS audit,comment,references + +# Enable this directive if you have tables or column names that are a reserved +# word for PostgreSQL. Ora2Pg will double quote the name of the object. +USE_RESERVED_WORDS 0 + +# By default Ora2Pg export Oracle tables with the NOLOGGING attribute as +# UNLOGGED tables. You may want to fully disable this feature because +# you will lost all data from unlogged table in case of PostgreSQL crash. +# Set it to 1 to export all tables as normal table. +DISABLE_UNLOGGED 0 + +#------------------------------------------------------------------------------ +# OUTPUT SECTION (Control output to file or PostgreSQL database) +#------------------------------------------------------------------------------ + +# Define the following directive to send export directly to a PostgreSQL +# database, this will disable file output. Note that these directives are only +# used for data export, other export need to be imported manually through the +# use og psql or any other PostgreSQL client. +#PG_DSN dbi:Pg:dbname=test_db;host=localhost;port=5432 +#PG_USER test +#PG_PWD test + +# By default all output is dump to STDOUT if not send directly to postgresql +# database (see above). Give a filename to save export to it. If you want +# a Gzip'd compressed file just add the extension .gz to the filename (you +# need perl module Compress::Zlib from CPAN). Add extension .bz2 to use Bzip2 +# compression. +OUTPUT output.sql + +# Base directory where all dumped files must be written +#OUTPUT_DIR /var/tmp + +# Path to the bzip2 program. See OUTPUT directive above. +BZIP2 $bzip2 + +# Allow object constraints to be saved in a separate file during schema export. +# The file will be named CONSTRAINTS_OUTPUT. Where OUTPUT is the value of the +# corresponding configuration directive. You can use .gz xor .bz2 extension to +# enable compression. Default is to save all data in the OUTPUT file. This +# directive is usable only with TABLE export type. +FILE_PER_CONSTRAINT 0 + +# Allow indexes to be saved in a separate file during schema export. The file +# will be named INDEXES_OUTPUT. Where OUTPUT is the value of the corresponding +# configuration directive. You can use the .gz, .xor, or .bz2 file extension to +# enable compression. Default is to save all data in the OUTPUT file. This +# directive is usable only with TABLE or TABLESPACE export type. With the +# TABLESPACE export, it is used to write "ALTER INDEX ... TABLESPACE ..." into +# a separate file named TBSP_INDEXES_OUTPUT that can be loaded at end of the +# migration after the indexes creation to move the indexes. +FILE_PER_INDEX 0 + +# Allow foreign key declaration to be saved in a separate file during +# schema export. By default foreign keys are exported into the main +# output file or in the CONSTRAINT_output.sql file. When enabled foreign +# keys will be exported into a file named FKEYS_output.sql +FILE_PER_FKEYS 0 + +# Allow data export to be saved in one file per table/view. The files +# will be named as tablename_OUTPUT. Where OUTPUT is the value of the +# corresponding configuration directive. You can use .gz xor .bz2 +# extension to enable compression. Default is to save all data in one +# file. This is usable only during INSERT or COPY export type. +FILE_PER_TABLE 0 + +# Allow function export to be saved in one file per function/procedure. +# The files will be named as funcname_OUTPUT. Where OUTPUT is the value +# of the corresponding configuration directive. You can use .gz xor .bz2 +# extension to enable compression. Default is to save all data in one +# file. It is usable during FUNCTION, PROCEDURE, TRIGGER and PACKAGE +# export type. +FILE_PER_FUNCTION 0 + +# By default Ora2Pg will force Perl to use utf8 I/O encoding. This is done through +# a call to the Perl pragma: +# +# use open ':utf8'; +# +# You can override this encoding by using the BINMODE directive, for example you +# can set it to :locale to use your locale or iso-8859-7, it will respectively use +# +# use open ':locale'; +# use open ':encoding(iso-8859-7)'; +# +# If you have change the NLS_LANG in non UTF8 encoding, you might want to set this +# directive. See http://perldoc.perl.org/5.14.2/open.html for more information. +# Most of the time, you might leave this directive commented. +#BINMODE utf8 + +# Set it to 0 to not include the call to \\set ON_ERROR_STOP ON in all SQL +# scripts. By default this order is always present. +STOP_ON_ERROR 1 + +# Enable this directive to use COPY FREEZE instead of a simple COPY to +# export data with rows already frozen. This is intended as a performance +# option for initial data loading. Rows will be frozen only if the table +# being loaded has been created or truncated in the current subtransaction. +# This will only works with export to file and when -J or ORACLE_COPIES is +# not set or default to 1. It can be used with direct import into PostgreSQL +# under the same condition but -j or JOBS must also be unset or default to 1. +COPY_FREEZE 0 + +# By default Ora2Pg use CREATE OR REPLACE in function DDL, if you need not +# to override existing functions disable this configuration directive, +# DDL will not include OR REPLACE. +CREATE_OR_REPLACE 1 + +# This directive can be used to send an initial command to PostgreSQL, just +# after the connection. For example to set some session parameters. This +# directive can be used multiple time. +#PG_INITIAL_COMMAND + + + +#------------------------------------------------------------------------------ +# TYPE SECTION (Control type behaviors and redefinitions) +#------------------------------------------------------------------------------ + +# If you're experiencing problems in data type export, the following directive +# will help you to redefine data type translation used in Ora2pg. The syntax is +# a comma separated list of "Oracle datatype:Postgresql data type". Here are the +# data type that can be redefined and their default value. If you want to +# replace a type with a precision and scale you need to escape the coma with +# a backslash. For example, if you want to replace all NUMBER(*,0) into bigint +# instead of numeric(38)add the following: +# DATA_TYPE NUMBER(*\\,0):bigint +# Here is the default replacement for all Oracle's types. You don't have to +# recopy all type conversion but just the one you want to rewrite. +#DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,DATE:timestamp,LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:numeric,INTEGER:numeric,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone + +# If set to 1 replace portable numeric type into PostgreSQL internal type. +# Oracle data type NUMBER(p,s) is approximatively converted to real and +# float PostgreSQL data type. If you have monetary fields or don't want +# rounding issues with the extra decimals you should preserve the same +# numeric(p,s) PostgreSQL data type. Do that only if you need exactness +# because using numeric(p,s) is slower than using real or double. +PG_NUMERIC_TYPE 1 + +# If set to 1 replace portable numeric type into PostgreSQL internal type. +# Oracle data type NUMBER(p) or NUMBER are converted to smallint, integer +# or bigint PostgreSQL data type following the length of the precision. If +# NUMBER without precision are set to DEFAULT_NUMERIC (see bellow). +PG_INTEGER_TYPE 1 + +# NUMBER() without precision are converted by default to bigint only if +# PG_INTEGER_TYPE is true. You can overwrite this value to any PG type, +# like integer or float. +DEFAULT_NUMERIC bigint + +# Set it to 0 if you don't want to export milliseconds from Oracle timestamp +# columns. Timestamp will be formated with to_char(..., 'YYYY-MM-DD HH24:MI:SS') +# Enabling this directive, the default, format is 'YYYY-MM-DD HH24:MI:SS.FF'. +ENABLE_MICROSECOND 1 + +# If you want to replace some columns as PostgreSQL boolean define here a list +# of tables and column separated by space as follows. You can also give a type +# and a precision to automatically convert all fields of that type as a boolean. +# For example: NUMBER:1 or CHAR:1 will replace any field of type number(1) or +# char(1) as a boolean in all exported tables. +#REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 + +# Use this to add additional definitions of the possible boolean values in Oracle +# field. You must set a space separated list of TRUE:FALSE values. BY default: +#BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled + +# When Ora2Pg find a "zero" date: 0000-00-00 00:00:00 it is replaced by a NULL. +# This could be a problem if your column is defined with NOT NULL constraint. +# If you can not remove the constraint, use this directive to set an arbitral +# date that will be used instead. You can also use -INFINITY if you don't want +# to use a fake date. +#REPLACE_ZERO_DATE 1970-01-01 00:00:00 + +# Some time you need to force the destination type, for example a column +# exported as timestamp by Ora2Pg can be forced into type date. Value is +# a comma-separated list of TABLE:COLUMN:TYPE structure. If you need to use +# comma or space inside type definition you will have to backslash them. +# +# MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\,6) +# +# Type of table1.col3 will be replaced by a varchar and table1.col4 by +# a decimal with precision and scale. +# +# If the column's type is a user defined type Ora2Pg will autodetect the +# composite type and will export its data using ROW(). Some Oracle user +# defined types are just array of a native type, in this case you may want +# to transform this column in simple array of a PostgreSQL native type. +# To do so, just redefine the destination type as wanted and Ora2Pg will +# also transform the data as an array. For example, with the following +# definition in Oracle: +# +# CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); +# CREATE TABLE club (Name VARCHAR2(10), +# Address VARCHAR2(20), +# City VARCHAR2(20), +# Phone VARCHAR2(8), +# Members mem_type +# ); +# +# custom type "mem_type" is just a string array and can be translated into +# the following in PostgreSQL: +# +# CREATE TABLE club ( +# name varchar(10), +# address varchar(20), +# city varchar(20), +# phone varchar(8), +# members text[] +# ) ; +# +# To do so, just use the directive as follow: +# +# MODIFY_TYPE CLUB:MEMBERS:text[] +# +# Ora2Pg will take care to transform all data of this column in the correct +# format. Only arrays of characters and numerics types are supported. +#MODIFY_TYPE + +# By default Oracle call to function TO_NUMBER will be translated as a cast +# into numeric. For example, TO_NUMBER('10.1234') is converted into PostgreSQL +# call to_number('10.1234')::numeric. If you want you can cast the call to integer +# or bigint by changing the value of the configuration directive. If you need +# better control of the format, just set it as value, for example: +# TO_NUMBER_CONVERSION 99999999999999999999.9999999999 +# will convert the code above as: +# TO_NUMBER('10.1234', '99999999999999999999.9999999999') +# Any value of the directive that it is not numeric, integer or bigint will +# be taken as a mask format. If set to none, no conversion will be done. +TO_NUMBER_CONVERSION numeric + +#------------------------------------------------------------------------------ +# GRANT SECTION (Control priviledge and owner export) +#------------------------------------------------------------------------------ + +# Set this to 1 to replace default password for all extracted user +# during GRANT export +GEN_USER_PWD 0 + +# By default the owner of database objects is the one you're using to connect +# to PostgreSQL. If you use an other user (e.g. postgres) you can force +# Ora2Pg to set the object owner to be the one used in the Oracle database by +# setting the directive to 1, or to a completely different username by setting +# the directive value # to that username. +FORCE_OWNER 0 + +# Ora2Pg use the function's security privileges set in Oracle and it is often +# defined as SECURITY DEFINER. If you want to override those security privileges +# for all functions and use SECURITY DEFINER instead, enable this directive. +FORCE_SECURITY_INVOKER 0 + +#------------------------------------------------------------------------------ +# DATA SECTION (Control data export behaviors) +#------------------------------------------------------------------------------ + +# Extract data by bulk of DATA_LIMIT tuples at once. Default 10000. If you set +# a high value be sure to have enough memory if you have million of rows. +DATA_LIMIT $DATA_LIMIT_DEFAULT + +# When Ora2Pg detect a table with some BLOB it will automatically reduce the +# value of this directive by dividing it by 10 until his value is below 1000. +# You can control this value by setting BLOB_LIMIT. Exporting BLOB use lot of +# ressources, setting it to a too high value can produce OOM. +#BLOB_LIMIT 500 + +# By default all data that are not of type date or time are escaped. If you +# experience any problem with that you can set it to 1 to disable it. This +# directive is only used during a COPY export type. +# See STANDARD_CONFORMING_STRINGS for enabling/disabling escape with INSERT +# statements. +NOESCAPE 0 + +# This directive may be used if you want to change the default isolation +# level of the data export transaction. Default is now to set the level +# to a serializable transaction to ensure data consistency. Here are the +# allowed value of this directive: readonly, readwrite, serializable and +# committed (read committed). +TRANSACTION serializable + +# This controls whether ordinary string literals ('...') treat backslashes +# literally, as specified in SQL standard. This was the default before Ora2Pg +# v8.5 so that all strings was escaped first, now this is currently on, causing +# Ora2Pg to use the escape string syntax (E'...') if this parameter is not +# set to 0. This is the exact behavior of the same option in PostgreSQL. +# This directive is only used during INSERT export to build INSERT statements. +# See NOESCAPE for enabling/disabling escape in COPY statements. +STANDARD_CONFORMING_STRINGS 1 + +# Use this directive to set the database handle's 'LongReadLen' attribute to +# a value that will be the larger than the expected size of the LOB. The default +# is 1MB witch may not be enough to extract BLOB objects. If the size of the LOB +# exceeds the 'LongReadLen' DBD::Oracle will return a 'ORA-24345: A Truncation' +# error. Default: 1023*1024 bytes. Take a look at this page to learn more: +# http://search.cpan.org/~pythian/DBD-Oracle-1.22/Oracle.pm#Data_Interface_for_Persistent_LOBs +# +# Important note: If you increase the value of this directive take care that +# DATA_LIMIT will probably needs to be reduced. Even if you only have a 1MB blob +# trying to read 10000 of them (the default DATA_LIMIT) all at once will require +# 10GB of memory. You may extract data from those table separately and set a +# DATA_LIMIT to 500 or lower, otherwise you may experience some out of memory. +#LONGREADLEN 1047552 + +# If you want to bypass the 'ORA-24345: A Truncation' error, set this directive +# to 1, it will truncate the data extracted to the LongReadLen value. +#LONGTRUNCOK 0 + +# Disable this if you want to load full content of BLOB and CLOB and not use +# LOB locators. In this case you will have to set LONGREADLEN to the right +# value. Note that this will not improve speed of BLOB export as most of the time is always +# consumed by the bytea escaping and in this case export is done line by line +# and not by chunk of DATA_LIMIT rows. For more information on how it works, see +# http://search.cpan.org/~pythian/DBD-Oracle-1.74/lib/DBD/Oracle.pm#Data_Interface_for_LOB_Locators +# Default is enabled, it use LOB locators. +USE_LOB_LOCATOR 1 + +# Oracle recommends reading from and writing to a LOB in batches using a +# multiple of the LOB chunk size. This chunk size defaults to 8k (8192). +# Recent tests shown that the best performances can be reach with higher +# value like 512K or 4Mb. +# +# A quick benchmark with 30120 rows with different size of BLOB (200x5Mb, +# 19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with DATA_LIMIT=100, +# LONGREADLEN=170Mb and a total table size of 20GB gives: +# +# no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) +# chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) +# chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) +# chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) +# +# In conclusion it can be more than 10 time faster with LOB_CHUNK_SIZE set +# to 4Mb. Dependind of the size of most BLOB you may want to adjust the value +# here. For example if you have a majority of small lobs bellow 8K, using 8192 +# is better to not waste space. +LOB_CHUNK_SIZE 512000 + +# Force the use of getStringVal() instead of getClobVal() for XML data export. +# Default is 1, enabled for backward compatibility. Set here to 0 to use extract +# method a la CLOB and export the XML code as it was stored. Note that XML value +# extracted with getStringVal() must not exceed VARCHAR2 size limit otherwize +# it will return an error. +XML_PRETTY 0 + +# Enable this directive if you want to continue direct data import on error. +# When Ora2Pg receives an error in the COPY or INSERT statement from PostgreSQL +# it will log the statement to a file called TABLENAME_error.log in the output +# directory and continue to next bulk of data. Like this you can try to fix the +# statement and manually reload the error log file. Default is disabled: abort +# import on error. +LOG_ON_ERROR 0 + +# If you want to convert CHAR(n) from Oracle into varchar(n) or text under +# PostgreSQL, you might want to do some triming on the data. By default +# Ora2Pg will auto-detect this conversion and remove any withspace at both +# leading and trailing position. If you just want to remove the leadings +# character, set the value to LEADING. If you just want to remove the trailing +# character, set the value to TRAILING. Default value is BOTH. +TRIM_TYPE BOTH + +# The default triming character is space, use the directive bellow if you need +# to change the character that will be removed. For example, set it to - if you +# have leading - in the char(n) field. To use space as triming charger, comment +# this directive, this is the default value. +#TRIM_CHAR - + +# Internal timestamp retrieves from custom type are extracted in the following +# format: 01-JAN-77 12.00.00.000000 AM. It is impossible to know the exact century +# that must be used, so by default any year below 49 will be added to 2000 +# and others to 1900. You can use this directive to change this default value. +# this is only relevant if you have user defined type with a column timestamp. +INTERNAL_DATE_MAX 49 + +# Disable this directive if you want to disable check_function_bodies. +# +# SET check_function_bodies = false; +# +# It disables validation of the function body string during CREATE FUNCTION. +# Default is to use de postgresql.conf setting that enable it by default. +FUNCTION_CHECK 1 + +# Exporting BLOB takes time, in some circumstances you may want to export +# all data except the BLOB columns. In this case disable this directive and +# the BLOB columns will not be included into data export. Take care that the +# target bytea column do not have a NOT NULL constraint. +ENABLE_BLOB_EXPORT 1 + +# By default data export order will be done by sorting on table name. If you +# have huge tables at end of alphabetic order and you are using multiprocess +# it can be better to set the sort order on size so that multiple small tables +# can be processed before the largest tables finish. In this case set this +# directive to size. Possible values are name and size. Note that export type +# SHOW_TABLE and SHOW_COLUMN will use this sort order too, not only COPY or +# INSERT export type. +DATA_EXPORT_ORDER name + +# By default Ora2Pg use \\i psql command to execute generated SQL files +# if you want to use a relative path following the script execution file +# enabling this option will use \\ir. See psql help for more information. +PSQL_RELATIVE_PATH 0 + + +#------------------------------------------------------------------------------ +# PERFORMANCES SECTION (Control export/import performances) +#------------------------------------------------------------------------------ + +# This configuration directive adds multiprocess support to COPY, FUNCTION +# and PROCEDURE export type, the value is the number of process to use. +# Default is to not use multiprocess. This directive is used to set the number +# of cores to used to parallelize data import into PostgreSQL. During FUNCTION +# or PROCEDURE export type each function will be translated to plpgsql using a +# new process, the performances gain can be very important when you have tons +# of function to convert. There's no more limitation in parallel processing +# than the number of cores and the PostgreSQL I/O performance capabilities. +# Doesn't works under Windows Operating System, it is simply disabled. +JOBS 1 + +# Multiprocess support. This directive should defined the number of parallel +# connection to Oracle when extracting data. The limit is the number of cores +# on your machine. This is useful if Oracle is the bottleneck. Take care that +# this directive can only be used if there is a column defined in DEFINED_PK. +ORACLE_COPIES 1 + +# Multiprocess support. This directive should defined the number of tables +# in parallel data extraction. The limit is the number of cores on your machine. +# Ora2Pg will open one database connection for each parallel table extraction. +# This directive, when upper than 1, will invalidate ORACLE_COPIES but not JOBS. +# Note that this directive when set upper that 1 will also automatically enable +# the FILE_PER_TABLE directive if your are exporting to files. +PARALLEL_TABLES 1 + +# You can force Ora2Pg to use /*+ PARALLEL(tbname, degree) */ hint in each +# query used to export data from Oracle by setting a value upper than 1 to +# this directive. A value of 0 or 1 disable the use of parallel hint. +# Default is disabled. +DEFAULT_PARALLELISM_DEGREE 0 + +# Parallel mode will not be activated if the table have less rows than +# this directive. This prevent fork of Oracle process when it is not +# necessary. Default is 100K rows. +PARALLEL_MIN_ROWS 100000 + +# Multiprocess support. This directive is used to split the select queries +# between the different connections to Oracle if ORA_COPIES is used. Ora2Pg +# will extract data with the following prepare statement: +# SELECT * FROM TABLE WHERE MOD(COLUMN, \$ORA_COPIES) = ? +# Where \$ORA_COPIES is the total number of cores used to extract data and set +# with ORA_COPIES directive, and ? is the current core used at execution time. +# This means that Ora2Pg needs to know the numeric column to use in this query. +# If this column is a real, float, numeric or decimal, you must add the ROUND() +# function with the column to round the value to the nearest integer. +#DEFINED_PK TABLE:COLUMN TABLE:ROUND(COLUMN) + +# Enabling this directive force Ora2Pg to drop all indexes on data import +# tables, except automatic index on primary key, and recreate them at end +# of data import. This may improve speed a lot during a fresh import. +DROP_INDEXES 0 + +# Specifies whether transaction commit will wait for WAL records to be written +# to disk before the command returns a "success" indication to the client. This +# is the equivalent to set synchronous_commit directive of postgresql.conf file. +# This is only used when you load data directly to PostgreSQL, the default is +# off to disable synchronous commit to gain speed at writing data. Some modified +# versions of PostgreSQL, like Greenplum, do not have this setting, so in this +# case set this directive to 1, ora2pg will not try to change the setting. +SYNCHRONOUS_COMMIT 0 + +#------------------------------------------------------------------------------ +# PLSQL SECTION (Control SQL and PL/SQL to PLPGSQL rewriting behaviors) +#------------------------------------------------------------------------------ + +# If the above configuration directive is not enough to validate your PL/SQL code +# enable this configuration directive to allow export of all PL/SQL code even if +# it is marked as invalid. The 'VALID' or 'INVALID' status applies to functions, +# procedures, packages and user defined types. +EXPORT_INVALID 0 + +# Enable PLSQL to PLPSQL conversion. This is a work in progress, feel +# free modify/add you own code and send me patches. The code is under +# function plsql_toplpgsql in Ora2PG/PLSQL.pm. Default enabled. +PLSQL_PGSQL 1 + +# Ora2Pg can replace all conditions with a test on NULL by a call to the +# coalesce() function to mimic the Oracle behavior where empty field are +# considered equal to NULL. Ex: (field1 IS NULL) and (field2 IS NOT NULL) will +# be replaced by (coalesce(field1::text, '') = '') and (field2 IS NOT NULL AND +# field2::text <> ''). You might want this replacement to be sure that your +# application will have the same behavior but if you have control on you app +# a better way is to change it to transform empty string into NULL because +# PostgreSQL makes the difference. +NULL_EQUAL_EMPTY 0 + +# Force empty_clob() and empty_blob() to be exported as NULL instead as empty +# string for the first one and \\\\x for the second. If NULL is allowed in your +# column this might improve data export speed if you have lot of empty lob. +EMPTY_LOB_NULL 1 + +# If you don't want to export package as schema but as simple functions you +# might also want to replace all call to package_name.function_name. If you +# disable the PACKAGE_AS_SCHEMA directive then Ora2Pg will replace all call +# to package_name.function_name() by package_name_function_name(). Default +# is to use a schema to emulate package. +PACKAGE_AS_SCHEMA 1 + +# Enable this directive if the rewrite of Oracle native syntax (+) of +# OUTER JOIN is broken. This will force Ora2Pg to not rewrite such code, +# default is to try to rewrite simple form of rigth outer join for the +# moment. +REWRITE_OUTER_JOIN 1 + +# By default Oracle functions are marked as STABLE as they can not modify data +# unless when used in PL/SQL with variable assignment or as conditional +# expression. You can force Ora2Pg to create these function as VOLATILE by +# disabling the following configuration directive. +FUNCTION_STABLE 1 + +# By default call to COMMIT/ROLLBACK are kept untouched by Ora2Pg to force +# the user to review the logic of the function. Once it is fixed in Oracle +# source code or you want to comment this calls enable the following directive +COMMENT_COMMIT_ROLLBACK 0 + +# It is common to see SAVEPOINT call inside PL/SQL procedure together with +# a ROLLBACK TO savepoint_name. When COMMENT_COMMIT_ROLLBACK is enabled you +# may want to also comment SAVEPOINT calls, in this case enable it. +COMMENT_SAVEPOINT 0 + +# Ora2Pg replace all string constant during the pl/sql to plpgsql translation, +# string constant are all text include between single quote. If you have some +# string placeholder used in dynamic call to queries you can set a list of +# regexp to be temporary replaced to not break the parser.The list of regexp +# must use the semi colon as separator. For exemple: +#STRING_CONSTANT_REGEXP + +# To support the Alternative Quoting Mechanism (''Q'') for String Literals +# set the regexp with the text capture to use to extract the text part. +# For example with a variable declared as +# c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; +# the regexp must be: q'{(.*)}' ora2pg use the \$\$ delimiter. +#ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' + +# If you want to use functions defined in the Orafce library and prevent +# Ora2Pg to translate call to these function, enable this directive. +# The Orafce library can be found here: https://github.com/orafce/orafce +# By default Ora2pg rewrite add_month(), add_year(), date_trunc() and +# to_char() functions, but you may prefer to use the orafce version of +# these function that do not need any code transformation. +USE_ORAFCE 0 + +# Enable translation of autonomous transactions into a wrapper function +# using dblink or pg_background extension. If you don't want to use this +# translation and just want the function to be exported as a normal one +# without the pragma call, disable this directive. +AUTONOMOUS_TRANSACTION 1 + +#------------------------------------------------------------------------------ +# ASSESSMENT SECTION (Control migration assessment behaviors) +#------------------------------------------------------------------------------ + +# Activate the migration cost evaluation. Must only be used with SHOW_REPORT, +# FUNCTION, PROCEDURE, PACKAGE and QUERY export type. Default is disabled. +# Note that enabling this directive will force PLSQL_PGSQL activation. +ESTIMATE_COST 0 + +# Set the value in minutes of the migration cost evaluation unit. Default +# is five minutes per unit. +COST_UNIT_VALUE 5 + +# By default when using SHOW_REPORT the migration report is generated as +# simple text, enabling this directive will force ora2pg to create a report +# in HTML format. +DUMP_AS_HTML 0 + +# Set the total number of tables to display in the Top N per row and size +# list in the SHOW_TABLE and SHOW_REPORT output. Default 10. +TOP_MAX 10 + +# Use this directive to redefined the number of human-days limit where the +# migration assessment level must switch from B to C. Default is set to 10 +# human-days. +HUMAN_DAYS_LIMIT 5 + +# Set the comma separated list of username that must be used to filter +# queries from the DBA_AUDIT_TRAIL table. Default is to not scan this +# table and to never look for queries. This parameter is used only with +# SHOW_REPORT and QUERY export type with no input file for queries. +# Note that queries will be normalized before output unlike when a file +# is given at input using the -i option or INPUT directive. +#AUDIT_USER USERNAME1,USERNAME2 + +# By default Ora2Pg will convert call to SYS_GUID() Oracle function +# with a call to uuid_generate_v4() from uuid-ossp extension. You can +# redefined it to use the gen_random_uuid() function from pgcrypto +# extension by changing the function name below. +#UUID_FUNCTION uuid_generate_v4 + +#------------------------------------------------------------------------------ +# POSTGRESQL FEATURE SECTION (Control which PostgreSQL features are available) +#------------------------------------------------------------------------------ + +# Set the PostgreSQL major version number of the target database. Ex: 9.6 or 10 +# Default is current major version at time of a new release. This replace the +# old PG_SUPPORTS_* configuration directives. +PG_VERSION 12 + +# Use btree_gin extenstion to create bitmap like index with pg >= 9.4 +# You will need to create the extension by yourself: +# create extension btree_gin; +# Default is to create GIN index, when disabled, a btree index will be created +BITMAP_AS_GIN 1 + +# Use pg_background extension to create an autonomous transaction instead +# of using a dblink wrapper. With pg >= 9.5 only, default is to use dblink. +PG_BACKGROUND 0 + +# By default if you have an autonomous transaction translated using dblink +# extension instead of pg_background the connection is defined using the +# values set with PG_DSN, PG_USER and PG_PWD. If you want to fully override +# the connection string use this directive as follow to set the connection +# in the autonomous transaction wrapper function. +#DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass + +# Some versions of PostgreSQL like Redshift doesn't support substr() +# and it need to be replaced by a call to substring(). In this case, +# disable it. +PG_SUPPORTS_SUBSTR 1 + +#------------------------------------------------------------------------------ +# SPATIAL SECTION (Control spatial geometry export) +#------------------------------------------------------------------------------ + +# Enable this directive if you want Ora2Pg to detect the real spatial type and +# dimensions used in a spatial column. By default Ora2Pg will look at spatial +# indexes to see if the layer_gtype and sdo_indx_dims constraint parameters have +# been set, otherwise column will be created with the non-constrained "geometry" +# type. Enabling this feature will force Ora2Pg to scan a sample of 50000 lines +# to look at the GTYPE used. You can increase or reduce the sample by setting +# the value of AUTODETECT_SPATIAL_TYPE to the desired number of line. +AUTODETECT_SPATIAL_TYPE 1 + +# Disable this directive if you don't want to automatically convert SRID to +# EPSG using the sdo_cs.map_oracle_srid_to_epsg() function. Default: enabled +# If the SDO_SRID returned by Oracle is NULL, it will be replaced by the +# default value 8307 converted to its EPSG value: 4326 (see DEFAULT_SRID) +# If the value is upper than 1, all SRID will be forced to this value, in +# this case DEFAULT_SRID will not be used when Oracle returns a null value +# and the value will be forced to CONVERT_SRID. +# Note that it is also possible to set the EPSG value on Oracle side when +# sdo_cs.map_oracle_srid_to_epsg() return NULL if your want to force the value: +# Ex: system> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; +CONVERT_SRID 1 + +# Use this directive to override the default EPSG SRID to used: 4326. +# Can be overwritten by CONVERT_SRID, see above. +DEFAULT_SRID 4326 + +# This directive can take three values: WKT (default), WKB and INTERNAL. +# When it is set to WKT, Ora2Pg will use SDO_UTIL.TO_WKTGEOMETRY() to +# extract the geometry data. When it is set to WKB, Ora2Pg will use the +# binary output using SDO_UTIL.TO_WKBGEOMETRY(). If those two extract type +# are called at Oracle side, they are slow and you can easily reach Out Of +# Memory when you have lot of rows. Also WKB is not able to export 3D geometry +# and some geometries like CURVEPOLYGON. In this case you may use the INTERNAL +# extraction type. It will use a pure Perl library to convert the SDO_GEOMETRY +# data into a WKT representation, the translation is done on Ora2Pg side. +# This is a work in progress, please validate your exported data geometries +# before use. +GEOMETRY_EXTRACT_TYPE INTERNAL + + +#------------------------------------------------------------------------------ +# FDW SECTION (Control Foreign Data Wrapper export) +#------------------------------------------------------------------------------ + +# This directive is used to set the name of the foreign data server that is used +# in the "CREATE SERVER name FOREIGN DATA WRAPPER oracle_fdw ..." command. This +# name will then be used in the "CREATE FOREIGN TABLE ..." SQL command. Default +# is arbitrary set to orcl. This only concerns export type FDW. +FDW_SERVER orcl + + +#------------------------------------------------------------------------------ +# MYSQL SECTION (Control MySQL export behavior) +#------------------------------------------------------------------------------ + +# Enable this if double pipe and double ampersand (|| and &&) should not be +# taken as equivalent to OR and AND. It depend of the variable \@sql_mode, +# Use it only if Ora2Pg fail on auto detecting this behavior. +MYSQL_PIPES_AS_CONCAT 0 + +# Enable this directive if you want EXTRACT() replacement to use the internal +# format returned as an integer, for example DD HH24:MM:SS will be replaced +# with format; DDHH24MMSS::bigint, this depend of your apps usage. +MYSQL_INTERNAL_EXTRACT_FORMAT 0 + +}; +close(OUTCFG); + +if ($^O !~ /MSWin32|dos/i) { + # Do not replace configuration directory in scripts/ora2pg if this is a RPM build. + if (!$ENV{RPM_BUILD_ROOT}) { + `perl -p -i -e 's#my \\\$CONFIG_FILE .*#my \\\$CONFIG_FILE = "$CONFDIR/ora2pg.conf";#' scripts/ora2pg`; + } else { + # Do not include prefix with rpmbuild + `perl -p -i -e 's#my \\\$CONFIG_FILE .*#my \\\$CONFIG_FILE = "$RPM_CONFDIR/ora2pg.conf";#' scripts/ora2pg`; + } +} else { + my $tmp_conf = quotemeta($CONFDIR); + `perl -p -e "s#my \\\$CONFIG_FILE .*#my \\\$CONFIG_FILE = '$tmp_conf\\\\ora2pg.conf';#" scripts\\ora2pg > scripts\\ora2pg.tmp`; + `copy scripts\\ora2pg.tmp scripts\\ora2pg /Y`; +} + +WriteMakefile( + 'NAME' => 'Ora2Pg', + 'VERSION_FROM' => 'lib/Ora2Pg.pm', + 'LICENSE' => 'GPLv3', + 'dist' => { + 'COMPRESS'=>'gzip -9f', 'SUFFIX' => 'gz', + 'ZIP'=>'/usr/bin/zip','ZIPFLAGS'=>'-rl' + }, + 'AUTHOR' => 'Gilles Darold (gilles _AT_ darold _DOT_ net)', + 'ABSTRACT' => 'Oracle to PostgreSQL migration toolkit', + 'EXE_FILES' => [ qw(scripts/ora2pg scripts/ora2pg_scanner) ], + 'MAN3PODS' => { 'doc/Ora2Pg.pod' => 'blib/man3/ora2pg.3' }, + 'DESTDIR' => $PREFIX, + 'INSTALLDIRS' => $ENV{INSTALLDIRS}, + 'clean' => {FILES => "$DEST_CONF_FILE lib/blib/"}, + 'PREREQ_PM' => {DBI => 0}, + 'META_MERGE' => { + resources => { + homepage => 'http://ora2pg.darold.net/', + repository => { + type => 'git', + git => 'git@github.com:darold/ora2pg.git', + web => 'https://github.com/darold/ora2pg', + }, + }, + } +); + +sub MY::install { + my $self = shift; + + my $string = $self->MM::install; + $string =~ s/(pure_install\s+)(.*)/$1 install_all $2/; + + return $string; +} + +sub MY::postamble { + my $postamble = qq{ +install_all : + \@echo "Installing default configuration file ($DEST_CONF_FILE) to $CONFDIR" + \@\$(MKPATH) $CONFDIR + \@\$(CP) -f $DEST_CONF_FILE $CONFDIR/$DEST_CONF_FILE + \@\$(MKPATH) $DOCDIR + \@\$(CP) -f README $DOCDIR/README + \@\$(CP) -f INSTALL $DOCDIR/INSTALL + \@\$(CP) -f changelog $DOCDIR/changelog +}; + if ($^O =~ /MSWin32|dos/i) { + my $tmp_conf = quotemeta($CONFDIR); + $postamble = qq{ +install_all : + \@echo "Installing default configuration file ($DEST_CONF_FILE) to $CONFDIR" + \@\$(MKPATH) $CONFDIR + \@\$(CP) $DEST_CONF_FILE $CONFDIR\\$DEST_CONF_FILE + \@\$(CP) README $CONFDIR\\README + \@\$(CP) INSTALL $CONFDIR\\INSTALL + \@\$(CP) changelog $CONFDIR\\changelog +}; + } + return $postamble; +} + + +if (!$ENV{QUIET}) { + print qq{ +Done... +------------------------------------------------------------------------------ +Please read documentation at http://ora2pg.darold.net/ before asking for help +------------------------------------------------------------------------------ +}; + if ($^O !~ /MSWin32|dos/i) { + print "Now type: make && make install\n"; + } else { + print "Now type: dmake && dmake install\n"; + } + +} diff --git a/README b/README new file mode 100644 index 0000000000000000000000000000000000000000..3d19fb93dd64f0625594c5a3b9ff69ed0c23c02a --- /dev/null +++ b/README @@ -0,0 +1,3039 @@ +NAME + Ora2Pg - Oracle to PostgreSQL database schema converter + +DESCRIPTION + Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL + compatible schema. It connects your Oracle database, scans it + automatically and extracts its structure or data, then generates SQL + scripts that you can load into your PostgreSQL database. + + Ora2Pg can be used for anything from reverse engineering Oracle database + to huge enterprise database migration or simply replicating some Oracle + data into a PostgreSQL database. It is really easy to use and doesn't + require any Oracle database knowledge other than providing the + parameters needed to connect to the Oracle database. + +FEATURES + Ora2Pg consist of a Perl script (ora2pg) and a Perl module (Ora2Pg.pm), + the only thing you have to modify is the configuration file ora2pg.conf + by setting the DSN to the Oracle database and optionally the name of a + schema. Once that's done you just have to set the type of export you + want: TABLE with constraints, VIEW, MVIEW, TABLESPACE, SEQUENCE, + INDEXES, TRIGGER, GRANT, FUNCTION, PROCEDURE, PACKAGE, PARTITION, TYPE, + INSERT or COPY, FDW, QUERY, KETTLE, SYNONYM. + + By default Ora2Pg exports to a file that you can load into PostgreSQL + with the psql client, but you can also import directly into a PostgreSQL + database by setting its DSN into the configuration file. With all + configuration options of ora2pg.conf you have full control of what + should be exported and how. + + Features included: + + - Export full database schema (tables, views, sequences, indexes), with + unique, primary, foreign key and check constraints. + - Export grants/privileges for users and groups. + - Export range/list partitions and sub partitions. + - Export a table selection (by specifying the table names). + - Export Oracle schema to a PostgreSQL 8.4+ schema. + - Export predefined functions, triggers, procedures, packages and + package bodies. + - Export full data or following a WHERE clause. + - Full support of Oracle BLOB object as PG BYTEA. + - Export Oracle views as PG tables. + - Export Oracle user defined types. + - Provide some basic automatic conversion of PLSQL code to PLPGSQL. + - Works on any platform. + - Export Oracle tables as foreign data wrapper tables. + - Export materialized view. + - Show a report of an Oracle database content. + - Migration cost assessment of an Oracle database. + - Migration difficulty level assessment of an Oracle database. + - Migration cost assessment of PL/SQL code from a file. + - Migration cost assessment of Oracle SQL queries stored in a file. + - Generate XML ktr files to be used with Penthalo Data Integrator (Kettle) + - Export Oracle locator and spatial geometries into PostGis. + - Export DBLINK as Oracle FDW. + - Export SYNONYMS as views. + - Export DIRECTORY as external table or directory for external_file extension. + - Full MySQL export just like Oracle database. + - Dispatch a list of SQL orders over multiple PostgreSQL connections + - Perform a diff between Oracle and PostgreSQL database for test purpose. + + Ora2Pg does its best to automatically convert your Oracle database to + PostgreSQL but there's still manual works to do. The Oracle specific + PL/SQL code generated for functions, procedures, packages and triggers + has to be reviewed to match the PostgreSQL syntax. You will find some + useful recommendations on porting Oracle PL/SQL code to PostgreSQL + PL/PGSQL at "Converting from other Databases to PostgreSQL", section: + Oracle (http://wiki.postgresql.org/wiki/Main_Page). + + See http://ora2pg.darold.net/report.html for a HTML sample of an Oracle + database migration report. + +INSTALLATION + All Perl modules can always be found at CPAN (http://search.cpan.org/). + Just type the full name of the module (ex: DBD::Oracle) into the search + input box, it will brings you the page for download. + + Releases of Ora2Pg stay at SF.net + (https://sourceforge.net/projects/ora2pg/). + + Under Windows you should install Strawberry Perl + (http://strawberryperl.com/) and the OSes corresponding Oracle clients. + Since version 5.32 this Perl distribution include pre-compiled driver of + DBD::Oracle and DBD::Pg. + + Requirement + The Oracle Instant Client or a full Oracle installation must be + installed on the system. You can download the RPM from Oracle download + center: + + rpm -ivh oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm + rpm -ivh oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm + rpm -ivh oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm + rpm -ivh oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm + + or simply download the corresponding ZIP archives from Oracle download + center and install them where you want, for example: + /opt/oracle/instantclient_12_2/ + + You also need a modern Perl distribution (perl 5.10 and more). To + connect to a database and proceed to his migration you need the DBI Perl + module > 1.614. To migrate an Oracle database you need the DBD::Oracle + Perl modules to be installed. To migrate a MySQL database you need the + DBD::MySQL Perl modules. These modules are used to connect to the + database but they are not mandatory if you want to migrate DDL input + files. + + To install DBD::Oracle and have it working you need to have the Oracle + client libraries installed and the ORACLE_HOME environment variable must + be defined. + + If you plan to export a MySQL database you need to install the Perl + module DBD::mysql which requires that the mysql client libraries are + installed. + + On some Perl distribution you may need to install the Time::HiRes Perl + module. + + If your distribution doesn't include these Perl modules you can install + them using CPAN: + + perl -MCPAN -e 'install DBD::Oracle' + perl -MCPAN -e 'install DBD::MySQL' + perl -MCPAN -e 'install Time::HiRes' + + otherwise use the packages provided by your distribution. + + Optional + By default Ora2Pg dumps export to flat files, to load them into your + PostgreSQL database you need the PostgreSQL client (psql). If you don't + have it on the host running Ora2Pg you can always transfer these files + to a host with the psql client installed. If you prefer to load export + 'on the fly', the perl module DBD::Pg is required. + + Ora2Pg allows you to dump all output in a compressed gzip file, to do + that you need the Compress::Zlib Perl module or if you prefer using + bzip2 compression, the program bzip2 must be available in your PATH. + + If your distribution doesn't include these Perl modules you can install + them using CPAN: + + perl -MCPAN -e 'install DBD::Pg' + perl -MCPAN -e 'install Compress::Zlib' + + otherwise use the packages provided by your distribution. + + Installing Ora2Pg + Like any other Perl Module Ora2Pg can be installed with the following + commands: + + tar xjf ora2pg-x.x.tar.bz2 + cd ora2pg-x.x/ + perl Makefile.PL + make && make install + + This will install Ora2Pg.pm into your site Perl repository, ora2pg into + /usr/local/bin/ and ora2pg.conf into /etc/ora2pg/. + + On Windows(tm) OSes you may use instead: + + perl Makefile.PL + dmake && dmake install + + This will install scripts and libraries into your Perl site installation + directory and the ora2pg.conf file as well as all documentation files + into C:\ora2pg\ + + To install ora2pg in a different directory than the default one, simply + use this command: + + perl Makefile.PL PREFIX= + make && make install + + then set PERL5LIB to the path to your installation directory before + using Ora2Pg. + + export PERL5LIB= + ora2pg -c config/ora2pg.conf -t TABLE -b outdir/ + + Packaging + If you want to build the binary package for your preferred Linux + distribution take a look at the packaging/ directory of the source + tarball. There is everything to build RPM, Slackware and Debian + packages. See README file in that directory. + + Installing DBD::Oracle + Ora2Pg needs the Perl module DBD::Oracle for connectivity to an Oracle + database from perl DBI. To get DBD::Oracle get it from CPAN a perl + module repository. + + After setting ORACLE_HOME and LD_LIBRARY_PATH environment variables as + root user, install DBD::Oracle. Proceed as follow: + + export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib + export ORACLE_HOME=/usr/lib/oracle/12.2/client64 + perl -MCPAN -e 'install DBD::Oracle' + + If you are running for the first time it will ask many questions; you + can keep defaults by pressing ENTER key, but you need to give one + appropriate mirror site for CPAN to download the modules. Install + through CPAN manually if the above doesn't work: + + #perl -MCPAN -e shell + cpan> get DBD::Oracle + cpan> quit + cd ~/.cpan/build/DBD-Oracle* + export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib + export ORACLE_HOME=/usr/lib/oracle/11.2/client64 + perl Makefile.PL + make + make install + + Installing DBD::Oracle require that the three Oracle packages: + instant-client, SDK and SQLplus are installed as well as the libaio1 + library. + + If you are using Instant Client from ZIP archives, the LD_LIBRARY_PATH + and ORACLE_HOME will be the same and must be set to the directory where + you have installed the files. For example: + /opt/oracle/instantclient_12_2/ + +CONFIGURATION + Ora2Pg configuration can be as simple as choosing the Oracle database to + export and choose the export type. This can be done in a minute. + + By reading this documentation you will also be able to: + + - Select only certain tables and/or column for export. + - Rename some tables and/or column during export. + - Select data to export following a WHERE clause per table. + - Delay database constraints during data loading. + - Compress exported data to save disk space. + - and much more. + + The full control of the Oracle database migration is taken though a + single configuration file named ora2pg.conf. The format of this file + consist in a directive name in upper case followed by tab character and + a value. Comments are lines beginning with a #. + + There's no specific order to place the configuration directives, they + are set at the time they are read in the configuration file. + + For configuration directives that just take a single value, you can use + them multiple time in the configuration file but only the last + occurrence found in the file will be used. For configuration directives + that allow a list of value, you can use it multiple time, the values + will be appended to the list. If you use the IMPORT directive to load a + custom configuration file, directives defined in this file will be + stores from the place the IMPORT directive is found, so it is better to + put it at the end of the configuration file. + + Values set in command line options will override values from the + configuration file. + + Ora2Pg usage + First of all be sure that libraries and binaries path include the Oracle + Instant Client installation: + + export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib + export PATH="/usr/lib/oracle/11.2/client64/bin:$PATH" + + By default Ora2Pg will look for /etc/ora2pg/ora2pg.conf configuration + file, if the file exist you can simply execute: + + /usr/local/bin/ora2pg + + or under Windows(tm) run ora2pg.bat file, located in your perl bin + directory. Windows(tm) users may also find a template configuration file + in C:\ora2pg + + If you want to call another configuration file, just give the path as + command line argument: + + /usr/local/bin/ora2pg -c /etc/ora2pg/new_ora2pg.conf + + Here are all command line parameters available when using ora2pg: + + Usage: ora2pg [-dhpqv --estimate_cost --dump_as_html] [--option value] + + -a | --allow str : Comma separated list of objects to allow from export. + Can be used with SHOW_COLUMN too. + -b | --basedir dir: Set the default output directory, where files + resulting from exports will be stored. + -c | --conf file : Set an alternate configuration file other than the + default /etc/ora2pg/ora2pg.conf. + -d | --debug : Enable verbose output. + -D | --data_type STR : Allow custom type replacement at command line. + -e | --exclude str: Comma separated list of objects to exclude from export. + Can be used with SHOW_COLUMN too. + -h | --help : Print this short help. + -g | --grant_object type : Extract privilege from the given object type. + See possible values with GRANT_OBJECT configuration. + -i | --input file : File containing Oracle PL/SQL code to convert with + no Oracle database connection initiated. + -j | --jobs num : Number of parallel process to send data to PostgreSQL. + -J | --copies num : Number of parallel connections to extract data from Oracle. + -l | --log file : Set a log file. Default is stdout. + -L | --limit num : Number of tuples extracted from Oracle and stored in + memory before writing, default: 10000. + -m | --mysql : Export a MySQL database instead of an Oracle schema. + -n | --namespace schema : Set the Oracle schema to extract from. + -N | --pg_schema schema : Set PostgreSQL's search_path. + -o | --out file : Set the path to the output file where SQL will + be written. Default: output.sql in running directory. + -p | --plsql : Enable PLSQL to PLPGSQL code conversion. + -P | --parallel num: Number of parallel tables to extract at the same time. + -q | --quiet : Disable progress bar. + -r | --relative : use \ir instead of \i in the psql scripts generated. + -s | --source DSN : Allow to set the Oracle DBI datasource. + -t | --type export: Set the export type. It will override the one + given in the configuration file (TYPE). + -T | --temp_dir DIR: Set a distinct temporary directory when two + or more ora2pg are run in parallel. + -u | --user name : Set the Oracle database connection user. + ORA2PG_USER environment variable can be used instead. + -v | --version : Show Ora2Pg Version and exit. + -w | --password pwd : Set the password of the Oracle database user. + ORA2PG_PASSWD environment variable can be used instead. + --forceowner : Force ora2pg to set tables and sequences owner like in + Oracle database. If the value is set to a username this one + will be used as the objects owner. By default it's the user + used to connect to the Pg database that will be the owner. + --nls_lang code: Set the Oracle NLS_LANG client encoding. + --client_encoding code: Set the PostgreSQL client encoding. + --view_as_table str: Comma separated list of views to export as table. + --estimate_cost : Activate the migration cost evaluation with SHOW_REPORT + --cost_unit_value minutes: Number of minutes for a cost evaluation unit. + default: 5 minutes, corresponds to a migration conducted by a + PostgreSQL expert. Set it to 10 if this is your first migration. + --dump_as_html : Force ora2pg to dump report in HTML, used only with + SHOW_REPORT. Default is to dump report as simple text. + --dump_as_csv : As above but force ora2pg to dump report in CSV. + --dump_as_sheet : Report migration assessment with one CSV line per database. + --init_project NAME: Initialise a typical ora2pg project tree. Top directory + will be created under project base dir. + --project_base DIR : Define the base dir for ora2pg project trees. Default + is current directory. + --print_header : Used with --dump_as_sheet to print the CSV header + especially for the first run of ora2pg. + --human_days_limit num : Set the number of human-days limit where the migration + assessment level switch from B to C. Default is set to + 5 human-days. + --audit_user LIST : Comma separated list of usernames to filter queries in + the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT + and QUERY export type. + --pg_dsn DSN : Set the datasource to PostgreSQL for direct import. + --pg_user name : Set the PostgreSQL user to use. + --pg_pwd password : Set the PostgreSQL password to use. + --count_rows : Force ora2pg to perform a real row count in TEST action. + --no_header : Do not append Ora2Pg header to output file + --oracle_speed : Use to know at which speed Oracle is able to send + data. No data will be processed or written. + --ora2pg_speed : Use to know at which speed Ora2Pg is able to send + transformed data. Nothing will be written. + + See full documentation at http://ora2pg.darold.net/ for more help or see + manpage with 'man ora2pg'. + + ora2pg will return 0 on success, 1 on error. It will return 2 when a + child process has been interrupted and you've gotten the warning + message: "WARNING: an error occurs during data export. Please check + what's happen." Most of the time this is an OOM issue, first try + reducing DATA_LIMIT value. + + For developers, it is possible to add your own custom option(s) in the + Perl script ora2pg as any configuration directive from ora2pg.conf can + be passed in lower case to the new Ora2Pg object instance. See ora2pg + code on how to add your own option. + + Note that performance might be improved by updating stats on oracle: + + BEGIN + DBMS_STATS.GATHER_SCHEMA_STATS + DBMS_STATS.GATHER_DATABASE_STATS + DBMS_STATS.GATHER_DICTIONARY_STATS + END; + + Generate a migration template + The two options --project_base and --init_project when used indicate to + ora2pg that he has to create a project template with a work tree, a + configuration file and a script to export all objects from the Oracle + database. Here a sample of the command usage: + + ora2pg --project_base /app/migration/ --init_project test_project + Creating project test_project. + /app/migration/test_project/ + schema/ + dblinks/ + directories/ + functions/ + grants/ + mviews/ + packages/ + partitions/ + procedures/ + sequences/ + synonyms/ + tables/ + tablespaces/ + triggers/ + types/ + views/ + sources/ + functions/ + mviews/ + packages/ + partitions/ + procedures/ + triggers/ + types/ + views/ + data/ + config/ + reports/ + + Generating generic configuration file + Creating script export_schema.sh to automate all exports. + Creating script import_all.sh to automate all imports. + + It create a generic config file where you just have to define the Oracle + database connection and a shell script called export_schema.sh. The + sources/ directory will contains the Oracle code, the schema/ will + contains the code ported to PostgreSQL. The reports/ directory will + contains the html reports with the migration cost assessment. + + If you want to use your own default config file, use the -c option to + give the path to that file. Rename it with .dist suffix if you want + ora2pg to apply the generic configuration values otherwise, the + configuration file will be copied untouched. + + Once you have set the connection to the Oracle Database you can execute + the script export_schema.sh that will export all object type from your + Oracle database and output DDL files into the schema's subdirectories. + At end of the export it will give you the command to export data later + when the import of the schema will be done and verified. + + You can choose to load the DDL files generated manually or use the + second script import_all.sh to import those file interactively. If this + kind of migration is not something current for you it's recommended you + to use those scripts. + + Oracle database connection + There's 5 configuration directives to control the access to the Oracle + database. + + ORACLE_HOME + Used to set ORACLE_HOME environment variable to the Oracle libraries + required by the DBD::Oracle Perl module. + + ORACLE_DSN + This directive is used to set the data source name in the form + standard DBI DSN. For example: + + dbi:Oracle:host=oradb_host.myhost.com;sid=DB_SID;port=1521 + + or + + dbi:Oracle:DB_SID + + On 18c this could be for example: + + dbi:Oracle:host=192.168.1.29;service_name=pdb1;port=1521 + + for the second notation the SID should be declared in the well known + file $ORACLE_HOME/network/admin/tnsnames.ora or in the path given to + the TNS_ADMIN environment variable. + + For MySQL the DSN will lool like this: + + dbi:mysql:host=192.168.1.10;database=sakila;port=3306 + + the 'sid' part is replaced by 'database'. + + ORACLE_USER et ORACLE_PWD + These two directives are used to define the user and password for + the Oracle database connection. Note that if you can it is better to + login as Oracle super admin to avoid grants problem during the + database scan and be sure that nothing is missing. + + If you do not supply a credential with ORACLE_PWD and you have + installed the Term::ReadKey Perl module, Ora2Pg will ask for the + password interactively. If ORACLE_USER is not set it will be asked + interactively too. + + To connect to a local ORACLE instance with connections "as sysdba" + you have to set ORACLE_USER to "/" and an empty password. + + USER_GRANTS + Set this directive to 1 if you connect the Oracle database as simple + user and do not have enough grants to extract things from the + DBA_... tables. It will use tables ALL_... instead. + + Warning: if you use export type GRANT, you must set this + configuration option to 0 or it will not work. + + TRANSACTION + This directive may be used if you want to change the default + isolation level of the data export transaction. Default is now to + set the level to a serializable transaction to ensure data + consistency. The allowed values for this directive are: + + readonly: 'SET TRANSACTION READ ONLY', + readwrite: 'SET TRANSACTION READ WRITE', + serializable: 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' + committed: 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED', + + Releases before 6.2 used to set the isolation level to READ ONLY + transaction but in some case this was breaking data consistency so + now default is set to SERIALIZABLE. + + INPUT_FILE + This directive did not control the Oracle database connection or + unless it purely disables the use of any Oracle database by + accepting a file as argument. Set this directive to a file + containing PL/SQL Oracle Code like function, procedure or full + package body to prevent Ora2Pg from connecting to an Oracle database + and just apply his conversion tool to the content of the file. This + can be used with the most of export types: TABLE, TRIGGER, + PROCEDURE, VIEW, FUNCTION or PACKAGE, etc. + + ORA_INITIAL_COMMAND + This directive can be used to send an initial command to Oracle, + just after the connection. For example to unlock a policy before + reading objects or to set some session parameters. This directive + can be used multiple times. + + Data encryption with Oracle server + If your Oracle Client config file already includes the encryption + method, then DBD:Oracle uses those settings to encrypt the connection + while you extract the data. For example if you have configured the + Oracle Client config file (sqlnet.or or .sqlnet) with the following + information: + + # Configure encryption of connections to Oracle + SQLNET.ENCRYPTION_CLIENT = required + SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256, RC4_256) + SQLNET.CRYPTO_SEED = 'should be 10-70 random characters' + + Any tool that uses the Oracle client to talk to the database will be + encrypted if you setup session encryption like above. + + For example, Perl's DBI uses DBD-Oracle, which uses the Oracle client + for actually handling database communication. If the installation of + Oracle client used by Perl is setup to request encrypted connections, + then your Perl connection to an Oracle database will also be encrypted. + + Full details at + https://kb.berkeley.edu/jivekb/entry.jspa?externalID=1005 + + Testing connection + Once you have set the Oracle database DSN you can execute ora2pg to see + if it works: + + ora2pg -t SHOW_VERSION -c config/ora2pg.conf + + will show the Oracle database server version. Take some time here to + test your installation as most problems take place here, the other + configuration steps are more technical. + + Troubleshooting + If the output.sql file has not exported anything other than the Pg + transaction header and footer there's two possible reasons. The perl + script ora2pg dump an ORA-XXX error, that mean that your DSN or login + information are wrong, check the error and your settings and try again. + The perl script says nothing and the output file is empty: the user + lacks permission to extract something from the database. Try to connect + to Oracle as super user or take a look at directive USER_GRANTS above + and at next section, especially the SCHEMA directive. + + LOGFILE + By default all messages are sent to the standard output. If you give + a file path to that directive, all output will be appended to this + file. + + Oracle schema to export + The Oracle database export can be limited to a specific Schema or + Namespace, this can be mandatory following the database connection user. + + SCHEMA + This directive is used to set the schema name to use during export. + For example: + + SCHEMA APPS + + will extract objects associated to the APPS schema. + + When no schema name is provided and EXPORT_SCHEMA is enabled, Ora2Pg + will export all objects from all schema of the Oracle instance with + their names prefixed with the schema name. + + EXPORT_SCHEMA + By default the Oracle schema is not exported into the PostgreSQL + database and all objects are created under the default Pg namespace. + If you want to also export this schema and create all objects under + this namespace, set the EXPORT_SCHEMA directive to 1. This will set + the schema search_path at top of export SQL file to the schema name + set in the SCHEMA directive with the default pg_catalog schema. If + you want to change this path, use the directive PG_SCHEMA. + + CREATE_SCHEMA + Enable/disable the CREATE SCHEMA SQL order at starting of the output + file. It is enable by default and concern on TABLE export type. + + COMPILE_SCHEMA + By default Ora2Pg will only export valid PL/SQL code. You can force + Oracle to compile again the invalidated code to get a chance to have + it obtain the valid status and then be able to export it. + + Enable this directive to force Oracle to compile schema before + exporting code. When this directive is enabled and SCHEMA is set to + a specific schema name, only invalid objects in this schema will be + recompiled. If SCHEMA is not set then all schema will be recompiled. + To force recompile invalid object in a specific schema, set + COMPILE_SCHEMA to the schema name you want to recompile. + + This will ask to Oracle to validate the PL/SQL that could have been + invalidate after a export/import for example. The 'VALID' or + 'INVALID' status applies to functions, procedures, packages and user + defined types. + + EXPORT_INVALID + If the above configuration directive is not enough to validate your + PL/SQL code enable this configuration directive to allow export of + all PL/SQL code even if it is marked as invalid. The 'VALID' or + 'INVALID' status applies to functions, procedures, packages and user + defined types. + + PG_SCHEMA + Allow you to defined/force the PostgreSQL schema to use. By default + if you set EXPORT_SCHEMA to 1 the PostgreSQL search_path will be set + to the schema name exported set as value of the SCHEMA directive. + + The value can be a comma delimited list of schema name but not when + using TABLE export type because in this case it will generate the + CREATE SCHEMA statement and it doesn't support multiple schema name. + For example, if you set PG_SCHEMA to something like "user_schema, + public", the search path will be set like this: + + SET search_path = user_schema, public; + + forcing the use of an other schema (here user_schema) than the one + from Oracle schema set in the SCHEMA directive. + + You can also set the default search_path for the PostgreSQL user you + are using to connect to the destination database by using: + + ALTER ROLE username SET search_path TO user_schema, public; + + in this case you don't have to set PG_SCHEMA. + + SYSUSERS + Without explicit schema, Ora2Pg will export all objects that not + belongs to system schema or role: + + SYSTEM,CTXSYS,DBSNMP,EXFSYS,LBACSYS,MDSYS,MGMT_VIEW, + OLAPSYS,ORDDATA,OWBSYS,ORDPLUGINS,ORDSYS,OUTLN, + SI_INFORMTN_SCHEMA,SYS,SYSMAN,WK_TEST,WKSYS,WKPROXY, + WMSYS,XDB,APEX_PUBLIC_USER,DIP,FLOWS_020100,FLOWS_030000, + FLOWS_040100,FLOWS_010600,FLOWS_FILES,MDDATA,ORACLE_OCM, + SPATIAL_CSW_ADMIN_USR,SPATIAL_WFS_ADMIN_USR,XS$NULL,PERFSTAT, + SQLTXPLAIN,DMSYS,TSMSYS,WKSYS,APEX_040000,APEX_040200, + DVSYS,OJVMSYS,GSMADMIN_INTERNAL,APPQOSSYS,DVSYS,DVF, + AUDSYS,APEX_030200,MGMT_VIEW,ODM,ODM_MTR,TRACESRV,MTMSYS, + OWBSYS_AUDIT,WEBSYS,WK_PROXY,OSE$HTTP$ADMIN, + AURORA$JIS$UTILITY$,AURORA$ORB$UNAUTHENTICATED, + DBMS_PRIVILEGE_CAPTURE,CSMIG,MGDSYS,SDE,DBSFWUSER + + Following your Oracle installation you may have several other system + role defined. To append these users to the schema exclusion list, + just set the SYSUSERS configuration directive to a comma-separated + list of system user to exclude. For example: + + SYSUSERS INTERNAL,SYSDBA,BI,HR,IX,OE,PM,SH + + will add users INTERNAL and SYSDBA to the schema exclusion list. + + FORCE_OWNER + By default the owner of the database objects is the one you're using + to connect to PostgreSQL using the psql command. If you use an other + user (postgres for example) you can force Ora2Pg to set the object + owner to be the one used in the Oracle database by setting the + directive to 1, or to a completely different username by setting the + directive value to that username. + + FORCE_SECURITY_INVOKER + Ora2Pg use the function's security privileges set in Oracle and it + is often defined as SECURITY DEFINER. If you want to override those + security privileges for all functions and use SECURITY DEFINER + instead, enable this directive. + + USE_TABLESPACE + When enabled this directive force ora2pg to export all tables, + indexes constraint and indexes using the tablespace name defined in + Oracle database. This works only with tablespace that are not TEMP, + USERS and SYSTEM. + + WITH_OID + Activating this directive will force Ora2Pg to add WITH (OIDS) when + creating tables or views as tables. Default is same as PostgreSQL, + disabled. + + LOOK_FORWARD_FUNCTION + List of schema to get functions/procedures meta information that are + used in the current schema export. When replacing call to function + with OUT parameters, if a function is declared in an other package + then the function call rewriting can not be done because Ora2Pg only + knows about functions declared in the current schema. By setting a + comma separated list of schema as value of this directive, Ora2Pg + will look forward in these packages for all + functions/procedures/packages declaration before proceeding to + current schema export. + + NO_FUNCTION_METADATA + Force Ora2Pg to not look for function declaration. Note that this + will prevent Ora2Pg to rewrite function replacement call if needed. + Do not enable it unless looking forward at function breaks other + export. + + Export type + The export action is perform following a single configuration directive + 'TYPE', some other add more control on what should be really exported. + + TYPE + Here are the different values of the TYPE directive, default is + TABLE: + + - TABLE: Extract all tables with indexes, primary keys, unique keys, + foreign keys and check constraints. + - VIEW: Extract only views. + - GRANT: Extract roles converted to Pg groups, users and grants on all + objects. + - SEQUENCE: Extract all sequence and their last position. + - TABLESPACE: Extract storage spaces for tables and indexes (Pg >= v8). + - TRIGGER: Extract triggers defined following actions. + - FUNCTION: Extract functions. + - PROCEDURE: Extract procedures. + - PACKAGE: Extract packages and package bodies. + - INSERT: Extract data as INSERT statement. + - COPY: Extract data as COPY statement. + - PARTITION: Extract range and list Oracle partitions with subpartitions. + - TYPE: Extract user defined Oracle type. + - FDW: Export Oracle tables as foreign table for oracle_fdw. + - MVIEW: Export materialized view. + - QUERY: Try to automatically convert Oracle SQL queries. + - KETTLE: Generate XML ktr template files to be used by Kettle. + - DBLINK: Generate oracle foreign data wrapper server to use as dblink. + - SYNONYM: Export Oracle's synonyms as views on other schema's objects. + - DIRECTORY: Export Oracle's directories as external_file extension objects. + - LOAD: Dispatch a list of queries over multiple PostgreSQl connections. + - TEST: perform a diff between Oracle and PostgreSQL database. + - TEST_VIEW: perform a count on both side of rows returned by views + + Only one type of export can be perform at the same time so the TYPE + directive must be unique. If you have more than one only the last + found in the file will be registered. + + Some export type can not or should not be load directly into the + PostgreSQL database and still require little manual editing. This is + the case for GRANT, TABLESPACE, TRIGGER, FUNCTION, PROCEDURE, TYPE, + QUERY and PACKAGE export types especially if you have PLSQL code or + Oracle specific SQL in it. + + For TABLESPACE you must ensure that file path exist on the system + and for SYNONYM you may ensure that the object's owners and schemas + correspond to the new PostgreSQL database design. + + Note that you can chained multiple export by giving to the TYPE + directive a comma-separated list of export type, but in this case + you must not use COPY or INSERT with other export type. + + Ora2Pg will convert Oracle partition using table inheritance, + trigger and functions. See document at Pg site: + http://www.postgresql.org/docs/current/interactive/ddl-partitioning. + html + + The TYPE export allow export of user defined Oracle type. If you + don't use the --plsql command line parameter it simply dump Oracle + user type asis else Ora2Pg will try to convert it to PostgreSQL + syntax. + + The KETTLE export type requires that the Oracle and PostgreSQL DNS + are defined. + + Since Ora2Pg v8.1 there's three new export types: + + SHOW_VERSION : display Oracle version + SHOW_SCHEMA : display the list of schema available in the database. + SHOW_TABLE : display the list of tables available. + SHOW_COLUMN : display the list of tables columns available and the + Ora2PG conversion type from Oracle to PostgreSQL that will be + applied. It will also warn you if there's PostgreSQL reserved + words in Oracle object names. + + Here is an example of the SHOW_COLUMN output: + + [2] TABLE CURRENT_SCHEMA (1 rows) (Warning: 'CURRENT_SCHEMA' is a reserved word in PostgreSQL) + CONSTRAINT : NUMBER(22) => bigint (Warning: 'CONSTRAINT' is a reserved word in PostgreSQL) + FREEZE : VARCHAR2(25) => varchar(25) (Warning: 'FREEZE' is a reserved word in PostgreSQL) + ... + [6] TABLE LOCATIONS (23 rows) + LOCATION_ID : NUMBER(4) => smallint + STREET_ADDRESS : VARCHAR2(40) => varchar(40) + POSTAL_CODE : VARCHAR2(12) => varchar(12) + CITY : VARCHAR2(30) => varchar(30) + STATE_PROVINCE : VARCHAR2(25) => varchar(25) + COUNTRY_ID : CHAR(2) => char(2) + + Those extraction keywords are use to only display the requested + information and exit. This allows you to quickly know on what you + are going to work. + + The SHOW_COLUMN allow an other ora2pg command line option: '--allow + relname' or '-a relname' to limit the displayed information to the + given table. + + The SHOW_ENCODING export type will display the NLS_LANG and + CLIENT_ENCODING values that Ora2Pg will used and the real encoding + of the Oracle database with the corresponding client encoding that + could be used with PostgreSQL + + Since release v8.12, Ora2Pg allow you to export your Oracle Table + definition to be use with the oracle_fdw foreign data wrapper. By + using type FDW your Oracle tables will be exported as follow: + + CREATE FOREIGN TABLE oratab ( + id integer NOT NULL, + text character varying(30), + floating double precision NOT NULL + ) SERVER oradb OPTIONS (table 'ORATAB'); + + Now you can use the table like a regular PostgreSQL table. + + See http://pgxn.org/dist/oracle_fdw/ for more information on this + foreign data wrapper. + + Release 10 adds a new export type destined to evaluate the content + of the database to migrate, in terms of objects and cost to end the + migration: + + SHOW_REPORT : show a detailed report of the Oracle database content. + + Here is a sample of report: http://ora2pg.darold.net/report.html + + There also a more advanced report with migration cost. See the + dedicated chapter about Migration Cost Evaluation. + + ESTIMATE_COST + Activate the migration cost evaluation. Must only be used with + SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE and QUERY export type. + Default is disabled. You may want to use the --estimate_cost command + line option instead to activate this functionality. Note that + enabling this directive will force PLSQL_PGSQL activation. + + COST_UNIT_VALUE + Set the value in minutes of the migration cost evaluation unit. + Default is five minutes per unit. See --cost_unit_value to change + the unit value at command line. + + DUMP_AS_HTML + By default when using SHOW_REPORT the migration report is generated + as simple text, enabling this directive will force ora2pg to create + a report in HTML format. + + See http://ora2pg.darold.net/report.html for a sample report. + + HUMAN_DAYS_LIMIT + Use this directive to redefined the number of human-days limit where + the migration assessment level must switch from B to C. Default is + set to 10 human-days. + + JOBS + This configuration directive adds multiprocess support to COPY, + FUNCTION and PROCEDURE export type, the value is the number of + process to use. Default is multiprocess disable. + + This directive is used to set the number of cores to used to + parallelize data import into PostgreSQL. During FUNCTION or + PROCEDURE export type each function will be translated to plpgsql + using a new process, the performances gain can be very important + when you have tons of function to convert. + + There's no limitation in parallel processing than the number of + cores and the PostgreSQL I/O performance capabilities. + + Doesn't work under Windows Operating System, it is simply disabled. + + ORACLE_COPIES + This configuration directive adds multiprocess support to extract + data from Oracle. The value is the number of process to use to + parallelize the select query. Default is parallel query disable. + + The parallelism is built on splitting the query following of the + number of cores given as value to ORACLE_COPIES as follow: + + SELECT * FROM MYTABLE WHERE ABS(MOD(COLUMN, ORACLE_COPIES)) = CUR_PROC + + where COLUMN is a technical key like a primary or unique key where + split will be based and the current core used by the query + (CUR_PROC). + + Doesn't work under Windows Operating System, it is simply disabled. + + DEFINED_PK + This directive is used to defined the technical key to used to split + the query between number of cores set with the ORACLE_COPIES + variable. For example: + + DEFINED_PK EMPLOYEES:employee_id + + The parallel query that will be used supposing that -J or + ORACLE_COPIES is set to 8: + + SELECT * FROM EMPLOYEES WHERE ABS(MOD(employee_id, 8)) = N + + where N is the current process forked starting from 0. + + PARALLEL_TABLES + This directive is used to defined the number of tables that will be + processed in parallel for data extraction. The limit is the number + of cores on your machine. Ora2Pg will open one database connection + for each parallel table extraction. This directive, when upper than + 1, will invalidate ORACLE_COPIES but not JOBS, so the real number of + process that will be used is PARALLEL_TABLES * JOBS. + + Note that this directive when set upper that 1 will also + automatically enable the FILE_PER_TABLE directive if your are + exporting to files. + + DEFAULT_PARALLELISM_DEGREE + You can force Ora2Pg to use /*+ PARALLEL(tbname, degree) */ hint in + each query used to export data from Oracle by setting a value upper + than 1 to this directive. A value of 0 or 1 disable the use of + parallel hint. Default is disabled. + + FDW_SERVER + This directive is used to set the name of the foreign data server + that is used in the "CREATE SERVER name FOREIGN DATA WRAPPER + oracle_fdw ..." command. This name will then be used in the "CREATE + FOREIGN TABLE ..." SQL command. Default is arbitrary set to orcl. + This only concern export type FDW. + + EXTERNAL_TO_FDW + This directive, enabled by default, allow to export Oracle's + External Tables as file_fdw foreign tables. To not export these + tables at all, set the directive to 0. + + INTERNAL_DATE_MAX + Internal timestamp retrieves from custom type are extracted in the + following format: 01-JAN-77 12.00.00.000000 AM. It is impossible to + know the exact century that must be used, so by default any year + below 49 will be added to 2000 and others to 1900. You can use this + directive to change the default value 49. this is only relevant if + you have user defined type with a column timestamp. + + AUDIT_USER + Set the comma separated list of username that must be used to filter + queries from the DBA_AUDIT_TRAIL table. Default is to not scan this + table and to never look for queries. This parameter is used only + with SHOW_REPORT and QUERY export type with no input file for + queries. Note that queries will be normalized before output unlike + when a file is given at input using the -i option or INPUT + directive. + + FUNCTION_CHECK + Disable this directive if you want to disable check_function_bodies. + + SET check_function_bodies = false; + + It disables validation of the function body string during CREATE + FUNCTION. Default is to use de postgresql.conf setting that enable + it by default. + + ENABLE_BLOB_EXPORT + Exporting BLOB takes time, in some circumstances you may want to + export all data except the BLOB columns. In this case disable this + directive and the BLOB columns will not be included into data + export. Take care that the target bytea column do not have a NOT + NULL constraint. + + DATA_EXPORT_ORDER + By default data export order will be done by sorting on table name. + If you have huge tables at end of alphabetic order and you are using + multiprocess, it can be better to set the sort order on size so that + multiple small tables can be processed before the largest tables + finish. In this case set this directive to size. Possible values are + name and size. Note that export type SHOW_TABLE and SHOW_COLUMN will + use this sort order too, not only COPY or INSERT export type. + + Limiting objects to export + You may want to export only a part of an Oracle database, here are a set + of configuration directives that will allow you to control what parts of + the database should be exported. + + ALLOW + This directive allows you to set a list of objects on which the + export must be limited, excluding all other objects in the same type + of export. The value is a space or comma-separated list of objects + name to export. You can include valid regex into the list. For + example: + + ALLOW EMPLOYEES SALE_.* COUNTRIES .*_GEOM_SEQ + + will export objects with name EMPLOYEES, COUNTRIES, all objects + beginning with 'SALE_' and all objects with a name ending by + '_GEOM_SEQ'. The object depends of the export type. Note that regex + will not works with 8i database, you must use the % placeholder + instead, Ora2Pg will use the LIKE operator. + + This is the manner to declare global filters that will be used with + the current export type. You can also use extended filters that will + be applied on specific objects or only on their related export type. + For example: + + ora2pg -p -c ora2pg.conf -t TRIGGER -a 'TABLE[employees]' + + will limit export of trigger to those defined on table employees. If + you want to extract all triggers but not some INSTEAD OF triggers: + + ora2pg -c ora2pg.conf -t TRIGGER -e 'VIEW[trg_view_.*]' + + Or a more complex form: + + ora2pg -p -c ora2pg.conf -t TABLE -a 'TABLE[EMPLOYEES]' \ + -e 'INDEX[emp_.*];CKEY[emp_salary_min]' + + This command will export the definition of the employee table but + will exclude all index beginning with 'emp_' and the CHECK + constraint called 'emp_salary_min'. + + When exporting partition you can exclude some partition tables by + using + + ora2pg -p -c ora2pg.conf -t PARTITION -e 'PARTITION[PART_199.* PART_198.*]' + + This will exclude partitioned tables for year 1980 to 1999 from the + export but not the main partition table. The trigger will also be + adapted to exclude those table. + + With GRANT export you can use this extended form to exclude some + users from the export or limit the export to some others: + + ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' + + or + + ora2pg -p -c ora2pg.conf -t GRANT -a 'GRANT[USER1 USER2]' + + will limit export grants to users USER1 and USER2. But if you don't + want to export grants on some functions for these users, for + example: + + ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' -e 'FUNCTION[adm_.*];PROCEDURE[adm_.*]' + + Advanced filters may need some learning. + + Oracle doesn't allow the use of lookahead expression so you may want + to exclude some object that match the ALLOW regexp you have defined. + For example if you want to export all table starting with E but not + those starting with EXP it is not possible to do that in a single + expression. This is why you can start a regular expression with the + ! character to exclude object matching the regexp given just after. + Our previous example can be written as follow: + + ALLOW E.* !EXP.* + + it will be translated into: + + REGEXP_LIKE(..., '^E.*$') AND NOT REGEXP_LIKE(..., '^EXP.*$') + + in the object search expression. + + EXCLUDE + This directive is the opposite of the previous, it allow you to + define a space or comma-separated list of object name to exclude + from the export. You can include valid regex into the list. For + example: + + EXCLUDE EMPLOYEES TMP_.* COUNTRIES + + will exclude object with name EMPLOYEES, COUNTRIES and all tables + beginning with 'tmp_'. + + For example, you can ban from export some unwanted function with + this directive: + + EXCLUDE write_to_.* send_mail_.* + + this example will exclude all functions, procedures or functions in + a package with the name beginning with those regex. Note that regex + will not work with 8i database, you must use the % placeholder + instead, Ora2Pg will use the NOT LIKE operator. + + See above (directive 'ALLOW') for the extended syntax. + + VIEW_AS_TABLE + Set which view to export as table. By default none. Value must be a + list of view name or regexp separated by space or comma. If the + object name is a view and the export type is TABLE, the view will be + exported as a create table statement. If export type is COPY or + INSERT, the corresponding data will be exported. + + See chapter "Exporting views as PostgreSQL table" for more details. + + NO_VIEW_ORDERING + By default Ora2Pg try to order views to avoid error at import time + with nested views. With a huge number of views this can take a very + long time, you can bypass this ordering by enabling this directive. + + GRANT_OBJECT + When exporting GRANTs you can specify a comma separated list of + objects for which privilege will be exported. Default is export for + all objects. Here are the possibles values TABLE, VIEW, MATERIALIZED + VIEW, SEQUENCE, PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, + DIRECTORY. Only one object type is allowed at a time. For example + set it to TABLE if you just want to export privilege on tables. You + can use the -g option to overwrite it. + + When used this directive prevent the export of users unless it is + set to USER. In this case only users definitions are exported. + + WHERE + This directive allows you to specify a WHERE clause filter when + dumping the contents of tables. Value is constructs as follows: + TABLE_NAME[WHERE_CLAUSE], or if you have only one where clause for + each table just put the where clause as the value. Both are possible + too. Here are some examples: + + # Global where clause applying to all tables included in the export + WHERE 1=1 + + # Apply the where clause only on table TABLE_NAME + WHERE TABLE_NAME[ID1='001'] + + # Applies two different clause on tables TABLE_NAME and OTHER_TABLE + # and a generic where clause on DATE_CREATE to all other tables + WHERE TABLE_NAME[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' OTHER_TABLE[NAME='test'] + + Any where clause not included into a table name bracket clause will + be applied to all exported table including the tables defined in the + where clause. These WHERE clauses are very useful if you want to + archive some data or at the opposite only export some recent data. + + To be able to quickly test data import it is useful to limit data + export to the first thousand tuples of each table. For Oracle define + the following clause: + + WHERE ROWNUM < 1000 + + and for MySQL, use the following: + + WHERE 1=1 LIMIT 1,1000 + + This can also be restricted to some tables data export. + + TOP_MAX + This directive is used to limit the number of item shown in the top + N lists like the top list of tables per number of rows and the top + list of largest tables in megabytes. By default it is set to 10 + items. + + LOG_ON_ERROR + Enable this directive if you want to continue direct data import on + error. When Ora2Pg received an error in the COPY or INSERT statement + from PostgreSQL it will log the statement to a file called + TABLENAME_error.log in the output directory and continue to next + bulk of data. Like this you can try to fix the statement and + manually reload the error log file. Default is disabled: abort + import on error. + + REPLACE_QUERY + Sometime you may want to extract data from an Oracle table but you + need a custom query for that. Not just a "SELECT * FROM table" like + Ora2Pg do but a more complex query. This directive allows you to + overwrite the query used by Ora2Pg to extract data. The format is + TABLENAME[SQL_QUERY]. If you have multiple table to extract by + replacing the Ora2Pg query, you can define multiple REPLACE_QUERY + lines. + + REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] + + Control of Full Text Search export + Several directives can be used to control the way Ora2Pg will export the + Oracle's Text search indexes. By default CONTEXT indexes will be + exported to PostgreSQL FTS indexes but CTXCAT indexes will be exported + as indexes using the pg_trgm extension. + + CONTEXT_AS_TRGM + Force Ora2Pg to translate Oracle Text indexes into PostgreSQL + indexes using pg_trgm extension. Default is to translate CONTEXT + indexes into FTS indexes and CTXCAT indexes using pg_trgm. Most of + the time using pg_trgm is enough, this is why this directive stand + for. You need to create the pg_trgm extension into the destination + database before importing the objects: + + CREATE EXTENSION pg_trgm; + + FTS_INDEX_ONLY + By default Ora2Pg creates a function-based index to translate Oracle + Text indexes. + + CREATE INDEX ON t_document + USING gin(to_tsvector('pg_catalog.french', title)); + + You will have to rewrite the CONTAIN() clause using to_tsvector(), + example: + + SELECT id,title FROM t_document + WHERE to_tsvector(title)) @@ to_tsquery('search_word'); + + To force Ora2Pg to create an extra tsvector column with a dedicated + triggers for FTS indexes, disable this directive. In this case, + Ora2Pg will add the column as follow: ALTER TABLE t_document ADD + COLUMN tsv_title tsvector; Then update the column to compute FTS + vectors if data have been loaded before UPDATE t_document SET + tsv_title = to_tsvector('pg_catalog.french', coalesce(title,'')); To + automatically update the column when a modification in the title + column appears, Ora2Pg adds the following trigger: + + CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS $$ + BEGIN + IF TG_OP = 'INSERT' OR new.title != old.title THEN + new.tsv_title := + to_tsvector('pg_catalog.french', coalesce(new.title,'')); + END IF; + return new; + END + $$ LANGUAGE plpgsql; + CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE + ON t_document + FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); + + When the Oracle text index is defined over multiple column, Ora2Pg + will use setweight() to set a weight in the order of the column + declaration. + + FTS_CONFIG + Use this directive to force text search configuration to use. When + it is not set, Ora2Pg will autodetect the stemmer used by Oracle for + each index and pg_catalog.english if the information is not found. + + USE_UNACCENT + If you want to perform your text search in an accent insensitive + way, enable this directive. Ora2Pg will create an helper function + over unaccent() and creates the pg_trgm indexes using this function. + With FTS Ora2Pg will redefine your text search configuration, for + example: + + CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); + ALTER TEXT SEARCH CONFIGURATION fr + ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; + + then set the FTS_CONFIG ora2pg.conf directive to fr instead of + pg_catalog.english. + + When enabled, Ora2pg will create the wrapper function: + + CREATE OR REPLACE FUNCTION unaccent_immutable(text) + RETURNS text AS + $$ + SELECT public.unaccent('public.unaccent', $1); + $$ LANGUAGE sql IMMUTABLE + COST 1; + + the indexes are exported as follow: + + CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document + USING gin (unaccent_immutable(title) gin_trgm_ops); + + In your queries you will need to use the same function in the search + to be able to use the function-based index. Example: + + SELECT * FROM t_document + WHERE unaccent_immutable(title) LIKE '%donnees%'; + + USE_LOWER_UNACCENT + Same as above but call lower() in the unaccent_immutable() function: + + CREATE OR REPLACE FUNCTION unaccent_immutable(text) + RETURNS text AS + $$ + SELECT lower(public.unaccent('public.unaccent', $1)); + $$ LANGUAGE sql IMMUTABLE; + + Modifying object structure + One of the great usage of Ora2Pg is its flexibility to replicate Oracle + database into PostgreSQL database with a different structure or schema. + There's three configuration directives that allow you to map those + differences. + + REORDERING_COLUMNS + Enable this directive to reordering columns and minimized the + footprint on disc, so that more rows fit on a data page, which is + the most important factor for speed. Default is disabled, that mean + the same order than in Oracle tables definition, that's should be + enough for most usage. This directive is only used with TABLE + export. + + MODIFY_STRUCT + This directive allows you to limit the columns to extract for a + given table. The value consist in a space-separated list of table + name with a set of column between parenthesis as follow: + + MODIFY_STRUCT NOM_TABLE(nomcol1,nomcol2,...) ... + + for example: + + MODIFY_STRUCT T_TEST1(id,dossier) T_TEST2(id,fichier) + + This will only extract columns 'id' and 'dossier' from table T_TEST1 + and columns 'id' and 'fichier' from the T_TEST2 table. This + directive can only be used with TABLE, COPY or INSERT export. With + TABLE export create table DDL will respect the new list of columns + and all indexes or foreign key pointing to or from a column removed + will not be exported. + + REPLACE_TABLES + This directive allows you to remap a list of Oracle table name to a + PostgreSQL table name during export. The value is a list of + space-separated values with the following structure: + + REPLACE_TABLES ORIG_TBNAME1:DEST_TBNAME1 ORIG_TBNAME2:DEST_TBNAME2 + + Oracle tables ORIG_TBNAME1 and ORIG_TBNAME2 will be respectively + renamed into DEST_TBNAME1 and DEST_TBNAME2 + + REPLACE_COLS + Like table name, the name of the column can be remapped to a + different name using the following syntax: + + REPLACE_COLS ORIG_TBNAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) + + For example: + + REPLACE_COLS T_TEST(dico:dictionary,dossier:folder) + + will rename Oracle columns 'dico' and 'dossier' from table T_TEST + into new name 'dictionary' and 'folder'. + + REPLACE_AS_BOOLEAN + If you want to change the type of some Oracle columns into + PostgreSQL boolean during the export you can define here a list of + tables and column separated by space as follow. + + REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 + + The values set in the boolean columns list will be replaced with the + 't' and 'f' following the default replacement values and those + additionally set in directive BOOLEAN_VALUES. + + Note that if you have modified the table name with REPLACE_TABLES + and/or the column's name, you need to use the name of the original + table and/or column. + + REPLACE_COLS TB_NAME1(OLD_COL_NAME1:NEW_COL_NAME1) + REPLACE_AS_BOOLEAN TB_NAME1:OLD_COL_NAME1 + + You can also give a type and a precision to automatically convert + all fields of that type as a boolean. For example: + + REPLACE_AS_BOOLEAN NUMBER:1 CHAR:1 TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 + + will also replace any field of type number(1) or char(1) as a + boolean in all exported tables. + + BOOLEAN_VALUES + Use this to add additional definition of the possible boolean values + used in Oracle fields. You must set a space-separated list of + TRUE:FALSE values. By default here are the values recognized by + Ora2Pg: + + BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled + + Any values defined here will be added to the default list. + + REPLACE_ZERO_DATE + When Ora2Pg find a "zero" date: 0000-00-00 00:00:00 it is replaced + by a NULL. This could be a problem if your column is defined with + NOT NULL constraint. If you can not remove the constraint, use this + directive to set an arbitral date that will be used instead. You can + also use -INFINITY if you don't want to use a fake date. + + INDEXES_SUFFIX + Add the given value as suffix to indexes names. Useful if you have + indexes with same name as tables. For example: + + INDEXES_SUFFIX _idx + + will add _idx at ed of all index name. Not so common but can help. + + INDEXES_RENAMING + Enable this directive to rename all indexes using + tablename_columns_names. Could be very useful for database that have + multiple time the same index name or that use the same name than a + table, which is not allowed by PostgreSQL Disabled by default. + + USE_INDEX_OPCLASS + Operator classes text_pattern_ops, varchar_pattern_ops, and + bpchar_pattern_ops support B-tree indexes on the corresponding + types. The difference from the default operator classes is that the + values are compared strictly character by character rather than + according to the locale-specific collation rules. This makes these + operator classes suitable for use by queries involving pattern + matching expressions (LIKE or POSIX regular expressions) when the + database does not use the standard "C" locale. If you enable, with + value 1, this will force Ora2Pg to export all indexes defined on + varchar2() and char() columns using those operators. If you set it + to a value greater than 1 it will only change indexes on columns + where the character limit is greater or equal than this value. For + example, set it to 128 to create these kind of indexes on columns of + type varchar2(N) where N >= 128. + + PREFIX_PARTITION + Enable this directive if you want that your partition table name + will be exported using the parent table name. Disabled by default. + If you have multiple partitioned table, when exported to PostgreSQL + some partitions could have the same name but different parent + tables. This is not allowed, table name must be unique. + + PREFIX_SUB_PARTITION + Enable this directive if you want that your subpartition table name + will be exported using the parent partition name. Enabled by + default. If the partition names are a part of the subpartition + names, you should enable this directive. + + DISABLE_PARTITION + If you don't want to reproduce the partitioning like in Oracle and + want to export all partitioned Oracle data into the main single + table in PostgreSQL enable this directive. Ora2Pg will export all + data into the main table name. Default is to use partitioning, + Ora2Pg will export data from each partition and import them into the + PostgreSQL dedicated partition table. + + DISABLE_UNLOGGED + By default Ora2Pg export Oracle tables with the NOLOGGING attribute + as UNLOGGED tables. You may want to fully disable this feature + because you will lose all data from unlogged tables in case of a + PostgreSQL crash. Set it to 1 to export all tables as normal tables. + + Oracle Spatial to PostGis + Ora2Pg fully export Spatial object from Oracle database. There's some + configuration directives that could be used to control the export. + + AUTODETECT_SPATIAL_TYPE + By default Ora2Pg is looking at indexes to see the spatial + constraint type and dimensions defined under Oracle. Those + constraints are passed as at index creation using for example: + + CREATE INDEX ... INDEXTYPE IS MDSYS.SPATIAL_INDEX + PARAMETERS('sdo_indx_dims=2, layer_gtype=point'); + + If those Oracle constraints parameters are not set, the default is + to export those columns as generic type GEOMETRY to be able to + receive any spatial type. + + The AUTODETECT_SPATIAL_TYPE directive allows to force Ora2Pg to + autodetect the real spatial type and dimension used in a spatial + column otherwise a non- constrained "geometry" type is used. + Enabling this feature will force Ora2Pg to scan a sample of 50000 + column to look at the GTYPE used. You can increase or reduce the + sample size by setting the value of AUTODETECT_SPATIAL_TYPE to the + desired number of line to scan. The directive is enabled by default. + + For example, in the case of a column named shape and defined with + Oracle type SDO_GEOMETRY, with AUTODETECT_SPATIAL_TYPE disabled it + will be converted as: + + shape geometry(GEOMETRY) or shape geometry(GEOMETRYZ, 4326) + + and if the directive is enabled and the column just contains a + single geometry type that use a single dimension: + + shape geometry(POLYGON, 4326) or shape geometry(POLYGONZ, 4326) + + with a two or three dimensional polygon. + + CONVERT_SRID + This directive allows you to control the automatically conversion of + Oracle SRID to standard EPSG. If enabled, Ora2Pg will use the Oracle + function sdo_cs.map_oracle_srid_to_epsg() to convert all SRID. + Enabled by default. + + If the SDO_SRID returned by Oracle is NULL, it will be replaced by + the default value 8307 converted to its EPSG value: 4326 (see + DEFAULT_SRID). + + If the value is upper than 1, all SRID will be forced to this value, + in this case DEFAULT_SRID will not be used when Oracle returns a + null value and the value will be forced to CONVERT_SRID. + + Note that it is also possible to set the EPSG value on Oracle side + when sdo_cs.map_oracle_srid_to_epsg() return NULL if your want to + force the value: + + system@db> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; + + DEFAULT_SRID + Use this directive to override the default EPSG SRID to used: 4326. + Can be overwritten by CONVERT_SRID, see above. + + GEOMETRY_EXTRACT_TYPE + This directive can take three values: WKT (default), WKB and + INTERNAL. When it is set to WKT, Ora2Pg will use + SDO_UTIL.TO_WKTGEOMETRY() to extract the geometry data. When it is + set to WKB, Ora2Pg will use the binary output using + SDO_UTIL.TO_WKBGEOMETRY(). If those two extract type are calls at + Oracle side, they are slow and you can easily reach Out Of Memory + when you have lot of rows. Also WKB is not able to export 3D + geometry and some geometries like CURVEPOLYGON. In this case you may + use the INTERNAL extraction type. It will use a Pure Perl library to + convert the SDO_GEOMETRY data into a WKT representation, the + translation is done on Ora2Pg side. This is a work in progress, + please validate your exported data geometries before use. Default + spatial object extraction type is INTERNAL. + + POSTGIS_SCHEMA + Use this directive to add a specific schema to the search path to + look for PostGis functions. + + PostgreSQL Import + By default conversion to PostgreSQL format is written to file + 'output.sql'. The command: + + psql mydb < output.sql + + will import content of file output.sql into PostgreSQL mydb database. + + DATA_LIMIT + When you are performing INSERT/COPY export Ora2Pg proceed by chunks + of DATA_LIMIT tuples for speed improvement. Tuples are stored in + memory before being written to disk, so if you want speed and have + enough system resources you can grow this limit to an upper value + for example: 100000 or 1000000. Before release 7.0 a value of 0 mean + no limit so that all tuples are stored in memory before being + flushed to disk. In 7.x branch this has been remove and chunk will + be set to the default: 10000 + + BLOB_LIMIT + When Ora2Pg detect a table with some BLOB it will automatically + reduce the value of this directive by dividing it by 10 until his + value is below 1000. You can control this value by setting + BLOB_LIMIT. Exporting BLOB use lot of resources, setting it to a too + high value can produce OOM. + + OUTPUT + The Ora2Pg output filename can be changed with this directive. + Default value is output.sql. if you set the file name with extension + .gz or .bz2 the output will be automatically compressed. This + require that the Compress::Zlib Perl module is installed if the + filename extension is .gz and that the bzip2 system command is + installed for the .bz2 extension. + + OUTPUT_DIR + Since release 7.0, you can define a base directory where the file + will be written. The directory must exists. + + BZIP2 + This directive allows you to specify the full path to the bzip2 + program if it can not be found in the PATH environment variable. + + FILE_PER_CONSTRAINT + Allow object constraints to be saved in a separate file during + schema export. The file will be named CONSTRAINTS_OUTPUT, where + OUTPUT is the value of the corresponding configuration directive. + You can use .gz xor .bz2 extension to enable compression. Default is + to save all data in the OUTPUT file. This directive is usable only + with TABLE export type. + + The constraints can be imported quickly into PostgreSQL using the + LOAD export type to parallelize their creation over multiple (-j or + JOBS) connections. + + FILE_PER_INDEX + Allow indexes to be saved in a separate file during schema export. + The file will be named INDEXES_OUTPUT, where OUTPUT is the value of + the corresponding configuration directive. You can use .gz xor .bz2 + file extension to enable compression. Default is to save all data in + the OUTPUT file. This directive is usable only with TABLE AND + TABLESPACE export type. With the TABLESPACE export, it is used to + write "ALTER INDEX ... TABLESPACE ..." into a separate file named + TBSP_INDEXES_OUTPUT that can be loaded at end of the migration after + the indexes creation to move the indexes. + + The indexes can be imported quickly into PostgreSQL using the LOAD + export type to parallelize their creation over multiple (-j or JOBS) + connections. + + FILE_PER_FKEYS + Allow foreign key declaration to be saved in a separate file during + schema export. By default foreign keys are exported into the main + output file or in the CONSTRAINT_output.sql file. When enabled + foreign keys will be exported into a file named FKEYS_output.sql + + FILE_PER_TABLE + Allow data export to be saved in one file per table/view. The files + will be named as tablename_OUTPUT, where OUTPUT is the value of the + corresponding configuration directive. You can still use .gz xor + .bz2 extension in the OUTPUT directive to enable compression. + Default 0 will save all data in one file, set it to 1 to enable this + feature. This is usable only during INSERT or COPY export type. + + FILE_PER_FUNCTION + Allow functions, procedures and triggers to be saved in one file per + object. The files will be named as objectname_OUTPUT. Where OUTPUT + is the value of the corresponding configuration directive. You can + still use .gz xor .bz2 extension in the OUTPUT directive to enable + compression. Default 0 will save all in one single file, set it to 1 + to enable this feature. This is usable only during the corresponding + export type, the package body export has a special behavior. + + When export type is PACKAGE and you've enabled this directive, + Ora2Pg will create a directory per package, named with the lower + case name of the package, and will create one file per + function/procedure into that directory. If the configuration + directive is not enabled, it will create one file per package as + packagename_OUTPUT, where OUTPUT is the value of the corresponding + directive. + + TRUNCATE_TABLE + If this directive is set to 1, a TRUNCATE TABLE instruction will be + add before loading data. This is usable only during INSERT or COPY + export type. + + When activated, the instruction will be added only if there's no + global DELETE clause or not one specific to the current table (see + below). + + DELETE + Support for include a DELETE FROM ... WHERE clause filter before + importing data and perform a delete of some lines instead of + truncating tables. Value is construct as follow: + TABLE_NAME[DELETE_WHERE_CLAUSE], or if you have only one where + clause for all tables just put the delete clause as single value. + Both are possible too. Here are some examples: + + DELETE 1=1 # Apply to all tables and delete all tuples + DELETE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST + DELETE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] + + The last applies two different delete where clause on tables + TABLE_TEST and TABLE_INFO and a generic delete where clause on + DATE_CREATE to all other tables. If TRUNCATE_TABLE is enabled it + will be applied to all tables not covered by the DELETE definition. + + These DELETE clauses might be useful with regular "updates". + + STOP_ON_ERROR + Set this parameter to 0 to not include the call to \set + ON_ERROR_STOP ON in all SQL scripts generated by Ora2Pg. By default + this order is always present so that the script will immediately + abort when an error is encountered. + + COPY_FREEZE + Enable this directive to use COPY FREEZE instead of a simple COPY to + export data with rows already frozen. This is intended as a + performance option for initial data loading. Rows will be frozen + only if the table being loaded has been created or truncated in the + current sub-transaction. This will only work with export to file and + when -J or ORACLE_COPIES is not set or default to 1. It can be used + with direct import into PostgreSQL under the same condition but -j + or JOBS must also be unset or default to 1. + + CREATE_OR_REPLACE + By default Ora2Pg uses CREATE OR REPLACE in function DDL, if you + need not to override existing functions disable this configuration + directive, DDL will not include OR REPLACE. + + NO_HEADER + Enabling this directive will prevent Ora2Pg to print his header into + output files. Only the translated code will be written. + + PSQL_RELATIVE_PATH + By default Ora2Pg use \i psql command to execute generated SQL files + if you want to use a relative path following the script execution + file enabling this option will use \ir. See psql help for more + information. + + When using Ora2Pg export type INSERT or COPY to dump data to file and + that FILE_PER_TABLE is enabled, you will be warned that Ora2Pg will not + export data again if the file already exists. This is to prevent + downloading twice table with huge amount of data. To force the download + of data from these tables you have to remove the existing output file + first. + + If you want to import data on the fly to the PostgreSQL database you + have three configuration directives to set the PostgreSQL database + connection. This is only possible with COPY or INSERT export type as for + database schema there's no real interest to do that. + + PG_DSN + Use this directive to set the PostgreSQL data source namespace using + DBD::Pg Perl module as follow: + + dbi:Pg:dbname=pgdb;host=localhost;port=5432 + + will connect to database 'pgdb' on localhost at tcp port 5432. + + Note that this directive is only used for data export, other export + need to be imported manually through the use og psql or any other + PostgreSQL client. + + PG_USER and PG_PWD + These two directives are used to set the login user and password. + + If you do not supply a credential with PG_PWD and you have installed + the Term::ReadKey Perl module, Ora2Pg will ask for the password + interactively. If PG_USER is not set it will be asked interactively + too. + + SYNCHRONOUS_COMMIT + Specifies whether transaction commit will wait for WAL records to be + written to disk before the command returns a "success" indication to + the client. This is the equivalent to set synchronous_commit + directive of postgresql.conf file. This is only used when you load + data directly to PostgreSQL, the default is off to disable + synchronous commit to gain speed at writing data. Some modified + version of PostgreSQL, like greenplum, do not have this setting, so + in this set this directive to 1, ora2pg will not try to change the + setting. + + PG_INITIAL_COMMAND + This directive can be used to send an initial command to PostgreSQL, + just after the connection. For example to set some session + parameters. This directive can be used multiple times. + + Column type control + PG_NUMERIC_TYPE + If set to 1 replace portable numeric type into PostgreSQL internal + type. Oracle data type NUMBER(p,s) is approximatively converted to + real and float PostgreSQL data type. If you have monetary fields or + don't want rounding issues with the extra decimals you should + preserve the same numeric(p,s) PostgreSQL data type. Do that only if + you need exactness because using numeric(p,s) is slower than using + real or double. + + PG_INTEGER_TYPE + If set to 1 replace portable numeric type into PostgreSQL internal + type. Oracle data type NUMBER(p) or NUMBER are converted to + smallint, integer or bigint PostgreSQL data type following the value + of the precision. If NUMBER without precision are set to + DEFAULT_NUMERIC (see below). + + DEFAULT_NUMERIC + NUMBER without precision are converted by default to bigint only if + PG_INTEGER_TYPE is true. You can overwrite this value to any PG + type, like integer or float. + + DATA_TYPE + If you're experiencing any problem in data type schema conversion + with this directive you can take full control of the correspondence + between Oracle and PostgreSQL types to redefine data type + translation used in Ora2pg. The syntax is a comma-separated list of + "Oracle datatype:Postgresql datatype". Here are the default list + used: + + DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,DATE:timestamp,LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:numeric,INTEGER:numeric,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone + + Note that the directive and the list definition must be a single + line. + + If you want to replace a type with a precision and scale you need to + escape the coma with a backslash. For example, if you want to + replace all NUMBER(*,0) into bigint instead of numeric(38) add the + following: + + DATA_TYPE NUMBER(*\,0):bigint + + You don't have to recopy all default type conversion but just the + one you want to rewrite. + + There's a special case with BFILE when they are converted to type + TEXT, they will just contains the full path to the external file. If + you set the destination type to BYTEA, the default, Ora2Pg will + export the content of the BFILE as bytea. The third case is when you + set the destination type to EFILE, in this case, Ora2Pg will export + it as an EFILE record: (DIRECTORY, FILENAME). Use the DIRECTORY + export type to export the existing directories as well as privileges + on those directories. + + There's no SQL function available to retrieve the path to the BFILE. + Ora2Pg have to create one using the DBMS_LOB package. + + CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) + RETURN VARCHAR2 + AS + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + l_path VARCHAR2(4000); + BEGIN + dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); + SELECT directory_path INTO l_path FROM all_directories + WHERE directory_name = l_dir; + l_dir := rtrim(l_path,'/'); + RETURN l_dir || '/' || l_fname; + END; + + This function is only created if Ora2Pg found a table with a BFILE + column and that the destination type is TEXT. The function is + dropped at the end of the export. This concern both, COPY and INSERT + export type. + + There's no SQL function available to retrieve BFILE as an EFILE + record, then Ora2Pg have to create one using the DBMS_LOB package. + + CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) + RETURN VARCHAR2 + AS + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + BEGIN + dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); + RETURN '(' || l_dir || ',' || l_fnamei || ')'; + END; + + This function is only created if Ora2Pg found a table with a BFILE + column and that the destination type is EFILE. The function is + dropped at the end of the export. This concern both, COPY and INSERT + export type. + + To set the destination type, use the DATA_TYPE configuration + directive: + + DATA_TYPE BFILE:EFILE + + for example. + + The EFILE type is a user defined type created by the PostgreSQL + extension external_file that can be found here: + https://github.com/darold/external_file This is a port of the BFILE + Oracle type to PostgreSQL. + + There's no SQL function available to retrieve the content of a + BFILE. Ora2Pg have to create one using the DBMS_LOB package. + + CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN + BLOB + AS + filecontent BLOB := NULL; + src_file BFILE := NULL; + l_step PLS_INTEGER := 12000; + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + offset NUMBER := 1; + BEGIN + IF p_bfile IS NULL THEN + RETURN NULL; + END IF; + + DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); + src_file := BFILENAME( l_dir, l_fname ); + IF src_file IS NULL THEN + RETURN NULL; + END IF; + + DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); + DBMS_LOB.CREATETEMPORARY(filecontent, true); + DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); + DBMS_LOB.FILECLOSE(src_file); + RETURN filecontent; + END; + + This function is only created if Ora2Pg found a table with a BFILE + column and that the destination type is bytea (the default). The + function is dropped at the end of the export. This concern both, + COPY and INSERT export type. + + About the ROWID and UROWID, they are converted into OID by "logical" + default but this will through an error at data import. There is no + equivalent data type so you might want to use the DATA_TYPE + directive to change the corresponding type in PostgreSQL. You should + consider replacing this data type by a bigserial (autoincremented + sequence), text or uuid data type. + + MODIFY_TYPE + Sometimes you need to force the destination type, for example a + column exported as timestamp by Ora2Pg can be forced into type date. + Value is a comma-separated list of TABLE:COLUMN:TYPE structure. If + you need to use comma or space inside type definition you will have + to backslash them. + + MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\,6) + + Type of table1.col3 will be replaced by a varchar and table1.col4 by + a decimal with precision and scale. + + If the column's type is a user defined type Ora2Pg will autodetect + the composite type and will export its data using ROW(). Some Oracle + user defined types are just array of a native type, in this case you + may want to transform this column in simple array of a PostgreSQL + native type. To do so, just redefine the destination type as wanted + and Ora2Pg will also transform the data as an array. For example, + with the following definition in Oracle: + + CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); + CREATE TABLE club (Name VARCHAR2(10), + Address VARCHAR2(20), + City VARCHAR2(20), + Phone VARCHAR2(8), + Members mem_type + ); + + custom type "mem_type" is just a string array and can be translated + into the following in PostgreSQL: + + CREATE TABLE club ( + name varchar(10), + address varchar(20), + city varchar(20), + phone varchar(8), + members text[] + ) ; + + To do so, just use the directive as follow: + + MODIFY_TYPE CLUB:MEMBERS:text[] + + Ora2Pg will take care to transform all data of this column in the + correct format. Only arrays of characters and numerics types are + supported. + + Taking export under control + The following other configuration directives interact directly with the + export process and give you fine granularity in database export control. + + SKIP + For TABLE export you may not want to export all schema constraints, + the SKIP configuration directive allows you to specify a + space-separated list of constraints that should not be exported. + Possible values are: + + - fkeys: turn off foreign key constraints + - pkeys: turn off primary keys + - ukeys: turn off unique column constraints + - indexes: turn off all other index types + - checks: turn off check constraints + + For example: + + SKIP indexes,checks + + will removed indexes and check constraints from export. + + PKEY_IN_CREATE + Enable this directive if you want to add primary key definition + inside the create table statement. If disabled (the default) primary + key definition will be added with an alter table statement. Enable + it if you are exporting to GreenPlum PostgreSQL database. + + KEEP_PKEY_NAMES + By default names of the primary and unique key in the source Oracle + database are ignored and key names are autogenerated in the target + PostgreSQL database with the PostgreSQL internal default naming + rules. If you want to preserve Oracle primary and unique key names + set this option to 1. + + FKEY_ADD_UPDATE + This directive allows you to add an ON UPDATE CASCADE option to a + foreign key when a ON DELETE CASCADE is defined or always. Oracle do + not support this feature, you have to use trigger to operate the ON + UPDATE CASCADE. As PostgreSQL has this feature, you can choose how + to add the foreign key option. There are three values to this + directive: never, the default that mean that foreign keys will be + declared exactly like in Oracle. The second value is delete, that + mean that the ON UPDATE CASCADE option will be added only if the ON + DELETE CASCADE is already defined on the foreign Keys. The last + value, always, will force all foreign keys to be defined using the + update option. + + FKEY_DEFERRABLE + When exporting tables, Ora2Pg normally exports constraints as they + are, if they are non-deferrable they are exported as non-deferrable. + However, non-deferrable constraints will probably cause problems + when attempting to import data to Pg. The FKEY_DEFERRABLE option set + to 1 will cause all foreign key constraints to be exported as + deferrable. + + DEFER_FKEY + In addition to exporting data when the DEFER_FKEY option set to 1, + it will add a command to defer all foreign key constraints during + data export and the import will be done in a single transaction. + This will work only if foreign keys have been exported as deferrable + and you are not using direct import to PostgreSQL (PG_DSN is not + defined). Constraints will then be checked at the end of the + transaction. + + This directive can also be enabled if you want to force all foreign + keys to be created as deferrable and initially deferred during + schema export (TABLE export type). + + DROP_FKEY + If deferring foreign keys is not possible due to the amount of data + in a single transaction, you've not exported foreign keys as + deferrable or you are using direct import to PostgreSQL, you can use + the DROP_FKEY directive. + + It will drop all foreign keys before all data import and recreate + them at the end of the import. + + DROP_INDEXES + This directive allows you to gain lot of speed improvement during + data import by removing all indexes that are not an automatic index + (indexes of primary keys) and recreate them at the end of data + import. Of course it is far better to not import indexes and + constraints before having imported all data. + + DISABLE_TRIGGERS + This directive is used to disable triggers on all tables in COPY or + INSERT export modes. Available values are USER (disable user-defined + triggers only) and ALL (includes RI system triggers). Default is 0: + do not add SQL statements to disable trigger before data import. + + If you want to disable triggers during data migration, set the value + to USER if your are connected as non superuser and ALL if you are + connected as PostgreSQL superuser. A value of 1 is equal to USER. + + DISABLE_SEQUENCE + If set to 1 it disables alter of sequences on all tables during COPY + or INSERT export mode. This is used to prevent the update of + sequence during data migration. Default is 0, alter sequences. + + NOESCAPE + By default all data that are not of type date or time are escaped. + If you experience any problem with that you can set it to 1 to + disable character escaping during data export. This directive is + only used during a COPY export. See STANDARD_CONFORMING_STRINGS for + enabling/disabling escape with INSERT statements. + + STANDARD_CONFORMING_STRINGS + This controls whether ordinary string literals ('...') treat + backslashes literally, as specified in SQL standard. This was the + default before Ora2Pg v8.5 so that all strings was escaped first, + now this is currently on, causing Ora2Pg to use the escape string + syntax (E'...') if this parameter is not set to 0. This is the exact + behavior of the same option in PostgreSQL. This directive is only + used during data export to build INSERT statements. See NOESCAPE for + enabling/disabling escape in COPY statements. + + TRIM_TYPE + If you want to convert CHAR(n) from Oracle into varchar(n) or text + on PostgreSQL using directive DATA_TYPE, you might want to do some + trimming on the data. By default Ora2Pg will auto-detect this + conversion and remove any whitespace at both leading and trailing + position. If you just want to remove the leadings character set the + value to LEADING. If you just want to remove the trailing character, + set the value to TRAILING. Default value is BOTH. + + TRIM_CHAR + The default trimming character is space, use this directive if you + need to change the character that will be removed. For example, set + it to - if you have leading - in the char(n) field. To use space as + trimming charger, comment this directive, this is the default value. + + PRESERVE_CASE + If you want to preserve the case of Oracle object name set this + directive to 1. By default Ora2Pg will convert all Oracle object + names to lower case. I do not recommend to enable this unless you + will always have to double-quote object names on all your SQL + scripts. + + ORA_RESERVED_WORDS + Allow escaping of column name using Oracle reserved words. Value is + a list of comma-separated reserved word. Default: + audit,comment,references. + + USE_RESERVED_WORDS + Enable this directive if you have table or column names that are a + reserved word for PostgreSQL. Ora2Pg will double quote the name of + the object. + + GEN_USER_PWD + Set this directive to 1 to replace default password by a random + password for all extracted user during a GRANT export. + + PG_SUPPORTS_MVIEW + Since PostgreSQL 9.3, materialized view are supported with the SQL + syntax 'CREATE MATERIALIZED VIEW'. To force Ora2Pg to use the native + PostgreSQL support you must enable this configuration - enable by + default. If you want to use the old style with table and a set of + function, you should disable it. + + PG_SUPPORTS_IFEXISTS + PostgreSQL version below 9.x do not support IF EXISTS in DDL + statements. Disabling the directive with value 0 will prevent Ora2Pg + to add those keywords in all generated statements. Default value is + 1, enabled. + + PG_SUPPORTS_ROLE (Deprecated) + This option is deprecated since Ora2Pg release v7.3. + + By default Oracle roles are translated into PostgreSQL groups. If + you have PostgreSQL 8.1 or more consider the use of ROLES and set + this directive to 1 to export roles. + + PG_SUPPORTS_INOUT (Deprecated) + This option is deprecated since Ora2Pg release v7.3. + + If set to 0, all IN, OUT or INOUT parameters will not be used into + the generated PostgreSQL function declarations (disable it for + PostgreSQL database version lower than 8.1), This is now enable by + default. + + PG_SUPPORTS_DEFAULT + This directive enable or disable the use of default parameter value + in function export. Until PostgreSQL 8.4 such a default value was + not supported, this feature is now enable by default. + + PG_SUPPORTS_WHEN (Deprecated) + Add support to WHEN clause on triggers as PostgreSQL v9.0 now + support it. This directive is enabled by default, set it to 0 + disable this feature. + + PG_SUPPORTS_INSTEADOF (Deprecated) + Add support to INSTEAD OF usage on triggers (used with PG >= 9.1), + if this directive is disabled the INSTEAD OF triggers will be + rewritten as Pg rules. + + PG_SUPPORTS_CHECKOPTION + When enabled, export views with CHECK OPTION. Disable it if you have + PostgreSQL version prior to 9.4. Default: 1, enabled. + + PG_SUPPORTS_IFEXISTS + If disabled, do not export object with IF EXISTS statements. Enabled + by default. + + PG_SUPPORTS_PARTITION + PostgreSQL version prior to 10.0 do not have native partitioning. + Enable this directive if you want to use declarative partitioning. + Enable by default. + + PG_SUPPORTS_SUBSTR + Some versions of PostgreSQL like Redshift doesn't support substr() + and it need to be replaced by a call to substring(). In this case, + disable it. + + PG_SUPPORTS_NAMED_OPERATOR + Disable this directive if you are using PG < 9.5, PL/SQL operator + used in named parameter => will be replaced by PostgreSQL + proprietary operator := Enable by default. + + PG_SUPPORTS_IDENTITY + Enable this directive if you have PostgreSQL >= 10 to use IDENTITY + columns instead of serial or bigserial data type. If + PG_SUPPORTS_IDENTITY is disabled and there is IDENTITY column in the + Oracle table, they are exported as serial or bigserial columns. When + it is enabled they are exported as IDENTITY columns like: + + CREATE TABLE identity_test_tab ( + id bigint GENERATED ALWAYS AS IDENTITY, + description varchar(30) + ) ; + + If there is non default sequence options set in Oracle, they will be + appended after the IDENTITY keyword. Additionally in both cases, + Ora2Pg will create a file AUTOINCREMENT_output.sql with a embedded + function to update the associated sequences with the restart value + set to "SELECT max(colname)+1 FROM tablename". Of course this file + must be imported after data import otherwise sequence will be kept + to start value. Enabled by default. + + PG_SUPPORTS_PROCEDURE + PostgreSQL v11 adds support of PROCEDURE, enable it if you use such + version. + + BITMAP_AS_GIN + Use btree_gin extension to create bitmap like index with pg >= 9.4 + You will need to create the extension by yourself: create extension + btree_gin; Default is to create GIN index, when disabled, a btree + index will be created + + PG_BACKGROUND + Use pg_background extension to create an autonomous transaction + instead of using a dblink wrapper. With pg >= 9.5 only. Default is + to use dblink. See https://github.com/vibhorkum/pg_background about + this extension. + + DBLINK_CONN + By default if you have an autonomous transaction translated using + dblink extension instead of pg_background the connection is defined + using the values set with PG_DSN, PG_USER and PG_PWD. If you want to + fully override the connection string use this directive as follow to + set the connection in the autonomous transaction wrapper function. + For example: + + DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass + + LONGREADLEN + Use this directive to set the database handle's 'LongReadLen' + attribute to a value that will be the larger than the expected size + of the LOBs. The default is 1MB witch may not be enough to extract + BLOBs or CLOBs. If the size of the LOB exceeds the 'LongReadLen' + DBD::Oracle will return a 'ORA-24345: A Truncation' error. Default: + 1023*1024 bytes. + + Take a look at this page to learn more: + http://search.cpan.org/~pythian/DBD-Oracle-1.22/Oracle.pm#Data_Inter + face_for_Persistent_LOBs + + Important note: If you increase the value of this directive take + care that DATA_LIMIT will probably needs to be reduced. Even if you + only have a 1MB blob, trying to read 10000 of them (the default + DATA_LIMIT) all at once will require 10GB of memory. You may extract + data from those table separately and set a DATA_LIMIT to 500 or + lower, otherwise you may experience some out of memory. + + LONGTRUNKOK + If you want to bypass the 'ORA-24345: A Truncation' error, set this + directive to 1, it will truncate the data extracted to the + LongReadLen value. Disable by default so that you will be warned if + your LongReadLen value is not high enough. + + USE_LOB_LOCATOR + Disable this if you want to load full content of BLOB and CLOB and + not use LOB locators. In this case you will have to set LONGREADLEN + to the right value. Note that this will not improve speed of BLOB + export as most of the time is always consumed by the bytea escaping + and in this case export is done line by line and not by chunk of + DATA_LIMIT rows. For more information on how it works, see + http://search.cpan.org/~pythian/DBD-Oracle-1.74/lib/DBD/Oracle.pm#Da + ta_Interface_for_LOB_Locators + + Default is enabled, it use LOB locators. + + LOB_CHUNK_SIZE + Oracle recommends reading from and writing to a LOB in batches using + a multiple of the LOB chunk size. This chunk size defaults to 8k + (8192). Recent tests shown that the best performances can be reach + with higher value like 512K or 4Mb. + + A quick benchmark with 30120 rows with different size of BLOB + (200x5Mb, 19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with + DATA_LIMIT=100, LONGREADLEN=170Mb and a total table size of 20GB + gives: + + no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) + chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) + chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) + chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) + + In conclusion it can be more than 10 time faster with LOB_CHUNK_SIZE + set to 4Mb. Depending of the size of most BLOB you may want to + adjust the value here. For example if you have a majority of small + lobs bellow 8K, using 8192 is better to not waste space. Default + value for LOB_CHUNK_SIZE is 512000. + + XML_PRETTY + Force the use getStringVal() instead of getClobVal() for XML data + export. Default is 1, enabled for backward compatibility. Set it to + 0 to use extract method a la CLOB. Note that XML value extracted + with getStringVal() must not exceed VARCHAR2 size limit (4000) + otherwise it will return an error. + + ENABLE_MICROSECOND + Set it to O if you want to disable export of millisecond from Oracle + timestamp columns. By default milliseconds are exported with the use + of following format: + + 'YYYY-MM-DD HH24:MI:SS.FF' + + Disabling will force the use of the following Oracle format: + + to_char(..., 'YYYY-MM-DD HH24:MI:SS') + + By default milliseconds are exported. + + DISABLE_COMMENT + Set this to 1 if you don't want to export comment associated to + tables and columns definition. Default is enabled. + + Control MySQL export behavior + MYSQL_PIPES_AS_CONCAT + Enable this if double pipe and double ampersand (|| and &&) should + not be taken as equivalent to OR and AND. It depend of the variable + @sql_mode, Use it only if Ora2Pg fail on auto detecting this + behavior. + + MYSQL_INTERNAL_EXTRACT_FORMAT + Enable this directive if you want EXTRACT() replacement to use the + internal format returned as an integer, for example DD HH24:MM:SS + will be replaced with format; DDHH24MMSS::bigint, this depend of + your apps usage. + + Special options to handle character encoding + NLS_LANG and NLS_NCHAR + By default Ora2Pg will set NLS_LANG to AMERICAN_AMERICA.AL32UTF8 and + NLS_NCHAR to AL32UTF8. It is not recommended to change those + settings but in some case it could be useful. Using your own + settings with those configuration directive will change the client + encoding at Oracle side by setting the environment variables + $ENV{NLS_LANG} and $ENV{NLS_NCHAR}. + + BINMODE + By default Ora2Pg will force Perl to use utf8 I/O encoding. This is + done through a call to the Perl pragma: + + use open ':utf8'; + + You can override this encoding by using the BINMODE directive, for + example you can set it to :locale to use your locale or iso-8859-7, + it will respectively use + + use open ':locale'; + use open ':encoding(iso-8859-7)'; + + If you have change the NLS_LANG in non UTF8 encoding, you might want + to set this directive. See http://perldoc.perl.org/5.14.2/open.html + for more information. Most of the time, leave this directive + commented. + + CLIENT_ENCODING + By default PostgreSQL client encoding is automatically set to UTF8 + to avoid encoding issue. If you have changed the value of NLS_LANG + you might have to change the encoding of the PostgreSQL client. + + You can take a look at the PostgreSQL supported character sets here: + http://www.postgresql.org/docs/9.0/static/multibyte.html + + PLSQL to PLPGSQL conversion + Automatic code conversion from Oracle PLSQL to PostgreSQL PLPGSQL is a + work in progress in Ora2Pg and surely you will always have manual work. + The Perl code used for automatic conversion is all stored in a specific + Perl Module named Ora2Pg/PLSQL.pm feel free to modify/add you own code + and send me patches. The main work in on function, procedure, package + and package body headers and parameters rewrite. + + PLSQL_PGSQL + Enable/disable PLSQL to PLPGSQL conversion. Enabled by default. + + NULL_EQUAL_EMPTY + Ora2Pg can replace all conditions with a test on NULL by a call to + the coalesce() function to mimic the Oracle behavior where empty + string are considered equal to NULL. + + (field1 IS NULL) is replaced by (coalesce(field1::text, '') = '') + (field2 IS NOT NULL) is replaced by (field2 IS NOT NULL AND field2::text <> '') + + You might want this replacement to be sure that your application + will have the same behavior but if you have control on you + application a better way is to change it to transform empty string + into NULL because PostgreSQL makes the difference. + + EMPTY_LOB_NULL + Force empty_clob() and empty_blob() to be exported as NULL instead + as empty string for the first one and '\x' for the second. If NULL + is allowed in your column this might improve data export speed if + you have lot of empty lob. Default is to preserve the exact data + from Oracle. + + PACKAGE_AS_SCHEMA + If you don't want to export package as schema but as simple + functions you might also want to replace all call to + package_name.function_name. If you disable the PACKAGE_AS_SCHEMA + directive then Ora2Pg will replace all call to + package_name.function_name() by package_name_function_name(). + Default is to use a schema to emulate package. + + The replacement will be done in all kind of DDL or code that is + parsed by the PLSQL to PLPGSQL converter. PLSQL_PGSQL must be + enabled or -p used in command line. + + REWRITE_OUTER_JOIN + Enable this directive if the rewrite of Oracle native syntax (+) of + OUTER JOIN is broken. This will force Ora2Pg to not rewrite such + code, default is to try to rewrite simple form of right outer join + for the moment. + + UUID_FUNCTION + By default Ora2Pg will convert call to SYS_GUID() Oracle function + with a call to uuid_generate_v4 from uuid-ossp extension. You can + redefined it to use the gen_random_uuid function from pgcrypto + extension by changing the function name. Default to + uuid_generate_v4. + + Note that when a RAW(n) column has "SYS_GUID()" as default value + Ora2Pg will automatically translate the type of the column into uuid + which might be the right translation in most of the case. + + FUNCTION_STABLE + By default Oracle functions are marked as STABLE as they can not + modify data unless when used in PL/SQL with variable assignment or + as conditional expression. You can force Ora2Pg to create these + function as VOLATILE by disabling this configuration directive. + + COMMENT_COMMIT_ROLLBACK + By default call to COMMIT/ROLLBACK are kept untouched by Ora2Pg to + force the user to review the logic of the function. Once it is fixed + in Oracle source code or you want to comment this calls enable the + following directive. + + COMMENT_SAVEPOINT + It is common to see SAVEPOINT call inside PL/SQL procedure together + with a ROLLBACK TO savepoint_name. When COMMENT_COMMIT_ROLLBACK is + enabled you may want to also comment SAVEPOINT calls, in this case + enable it. + + STRING_CONSTANT_REGEXP + Ora2Pg replace all string constant during the pl/sql to plpgsql + translation, string constant are all text include between single + quote. If you have some string placeholder used in dynamic call to + queries you can set a list of regexp to be temporary replaced to not + break the parser. For example: + + STRING_CONSTANT_REGEXP + + The list of regexp must use the semi colon as separator. + + ALTERNATIVE_QUOTING_REGEXP + To support the Alternative Quoting Mechanism ('Q' or 'q') for String + Literals set the regexp with the text capture to use to extract the + text part. For example with a variable declared as + + c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; + + the regexp to use must be: + + ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' + + ora2pg will use the $$ delimiter, with the example the result will + be: + + c_sample varchar(100) := $$This doesn't work.$$; + + The value of this configuration directive can be a list of regexp + separated by a semi colon. The capture part (between parenthesis) is + mandatory in each regexp if you want to restore the string constant. + + USE_ORAFCE + If you want to use functions defined in the Orafce library and + prevent Ora2Pg to translate call to these functions, enable this + directive. The Orafce library can be found here: + https://github.com/orafce/orafce + + By default Ora2pg rewrite add_month(), add_year(), date_trunc() and + to_char() functions, but you may prefer to use the orafce version of + these function that do not need any code transformation. + + AUTONOMOUS_TRANSACTION + Enable translation of autonomous transactions into a wrapper + function using dblink or pg_background extension. If you don't want + to use this translation and just want the function to be exported as + a normal one without the pragma call, disable this directive. + + Materialized view + Materialized views are exported as snapshot "Snapshot Materialized + Views" as PostgreSQL only supports full refresh. + + If you want to import the materialized views in PostgreSQL prior to 9.3 + you have to set configuration directive PG_SUPPORTS_MVIEW to 0. In this + case Ora2Pg will export all materialized views as explain in this + document: + + http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views. + + When exporting materialized view Ora2Pg will first add the SQL code to + create the "materialized_views" table: + + CREATE TABLE materialized_views ( + mview_name text NOT NULL PRIMARY KEY, + view_name text NOT NULL, + iname text, + last_refresh TIMESTAMP WITH TIME ZONE + ); + + all materialized views will have an entry in this table. It then adds + the plpgsql code to create tree functions: + + create_materialized_view(text, text, text) used to create a materialized view + drop_materialized_view(text) used to delete a materialized view + refresh_full_materialized_view(text) used to refresh a view + + then it adds the SQL code to create the view and the materialized view: + + CREATE VIEW mviewname_mview AS + SELECT ... FROM ...; + + SELECT create_materialized_view('mviewname','mviewname_mview', change with the name of the column to used for the index); + + The first argument is the name of the materialized view, the second the + name of the view on which the materialized view is based and the third + is the column name on which the index should be build (aka most of the + time the primary key). This column is not automatically deduced so you + need to replace its name. + + As said above Ora2Pg only supports snapshot materialized views so the + table will be entirely refreshed by issuing first a truncate of the + table and then by load again all data from the view: + + refresh_full_materialized_view('mviewname'); + + To drop the materialized view you just have to call the + drop_materialized_view() function with the name of the materialized view + as parameter. + + Other configuration directives + DEBUG + Set it to 1 will enable verbose output. + + IMPORT + You can define common Ora2Pg configuration directives into a single + file that can be imported into other configuration files with the + IMPORT configuration directive as follow: + + IMPORT commonfile.conf + + will import all configuration directives defined into + commonfile.conf into the current configuration file. + + Exporting views as PostgreSQL tables + You can export any Oracle view as a PostgreSQL table simply by setting + TYPE configuration option to TABLE to have the corresponding create + table statement. Or use type COPY or INSERT to export the corresponding + data. To allow that you have to specify your views in the VIEW_AS_TABLE + configuration option. + + Then if Ora2Pg finds the view it will extract its schema (if TYPE=TABLE) + into a PG create table form, then it will extract the data (if TYPE=COPY + or INSERT) following the view schema. + + For example, with the following view: + + CREATE OR REPLACE VIEW product_prices (category_id, product_count, low_price, high_price) AS + SELECT category_id, COUNT(*) as product_count, + MIN(list_price) as low_price, + MAX(list_price) as high_price + FROM product_information + GROUP BY category_id; + + Setting VIEW_AS_TABLE to product_prices and using export type TABLE, + will force Ora2Pg to detect columns returned types and to generate a + create table statement: + + CREATE TABLE product_prices ( + category_id bigint, + product_count integer, + low_price numeric, + high_price numeric + ); + + Data will be loaded following the COPY or INSERT export type and the + view declaration. + + You can use the ALLOW and EXCLUDE directive in addition to filter other + objects to export. + + Export as Kettle transformation XML files + The KETTLE export type is useful if you want to use Penthalo Data + Integrator (Kettle) to import data to PostgreSQL. With this type of + export Ora2Pg will generate one XML Kettle transformation files (.ktr) + per table and add a line to manually execute the transformation in the + output.sql file. For example: + + ora2pg -c ora2pg.conf -t KETTLE -j 12 -a MYTABLE -o load_mydata.sh + + will generate one file called 'HR.MYTABLE.ktr' and add a line to the + output file (load_mydata.sh): + + #!/bin/sh + + KETTLE_TEMPLATE_PATH='.' + + JAVAMAXMEM=4096 ./pan.sh -file $KETTLE_TEMPLATE_PATH/HR.MYTABLE.ktr -level Detailed + + The -j 12 option will create a template with 12 processes to insert data + into PostgreSQL. It is also possible to specify the number of parallel + queries used to extract data from the Oracle with the -J command line + option as follow: + + ora2pg -c ora2pg.conf -t KETTLE -J 4 -j 12 -a EMPLOYEES -o load_mydata.sh + + This is only possible if you have defined the technical key to used to + split the query between cores in the DEFINED_PKEY configuration + directive. For example: + + DEFINED_PK EMPLOYEES:employee_id + + will force the number of Oracle connection copies to 4 and defined the + SQL query as follow in the Kettle XML transformation file: + + SELECT * FROM HR.EMPLOYEES WHERE ABS(MOD(employee_id,${Internal.Step.Unique.Count}))=${Internal.Step.Unique.Number} + + The KETTLE export type requires that the Oracle and PostgreSQL DSN are + defined. You can also activate the TRUNCATE_TABLE directive to force a + truncation of the table before data import. + + The KETTLE export type is an original work of Marc Cousin. + + Migration cost assessment + Estimating the cost of a migration process from Oracle to PostgreSQL is + not easy. To obtain a good assessment of this migration cost, Ora2Pg + will inspect all database objects, all functions and stored procedures + to detect if there's still some objects and PL/SQL code that can not be + automatically converted by Ora2Pg. + + Ora2Pg has a content analysis mode that inspect the Oracle database to + generate a text report on what the Oracle database contains and what can + not be exported. + + To activate the "analysis and report" mode, you have to use the export + de type SHOW_REPORT like in the following command: + + ora2pg -t SHOW_REPORT + + Here is a sample report obtained with this command: + + -------------------------------------- + Ora2Pg: Oracle Database Content Report + -------------------------------------- + Version Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 + Schema HR + Size 880.00 MB + + -------------------------------------- + Object Number Invalid Comments + -------------------------------------- + CLUSTER 2 0 Clusters are not supported and will not be exported. + FUNCTION 40 0 Total size of function code: 81992. + INDEX 435 0 232 index(es) are concerned by the export, others are automatically generated and will + do so on PostgreSQL. 1 bitmap index(es). 230 b-tree index(es). 1 reversed b-tree index(es) + Note that bitmap index(es) will be exported as b-tree index(es) if any. Cluster, domain, + bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported + too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index + and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' + operators in your indexes to improve search with the LIKE operator respectively into + varchar, text or char columns. + MATERIALIZED VIEW 1 0 All materialized view will be exported as snapshot materialized views, they + are only updated when fully refreshed. + PACKAGE BODY 2 1 Total size of package code: 20700. + PROCEDURE 7 0 Total size of procedure code: 19198. + SEQUENCE 160 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL + will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). + TABLE 265 0 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration + directive to export as file_fdw foreign tables or use COPY in your code if you just + want to load data from external files. 2 binary columns. 4 unknown types. + TABLE PARTITION 8 0 Partitions are exported using table inheritance and check constraint. 1 HASH partitions. + 2 LIST partitions. 6 RANGE partitions. Note that Hash partitions are not supported. + TRIGGER 30 0 Total size of trigger code: 21677. + TYPE 7 1 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. + 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type + inherited and Subtype are converted as table, type inheritance is not supported. + TYPE BODY 0 3 Export of type with member method are not supported, they will not be exported. + VIEW 7 0 Views are fully supported, but if you have updatable views you will need to use + INSTEAD OF triggers. + DATABASE LINK 1 0 Database links will not be exported. You may try the dblink perl contrib module or use + the SQL/MED PostgreSQL features with the different Foreign Data Wrapper (FDW) extensions. + + Note: Invalid code will not be exported unless the EXPORT_INVALID configuration directive is activated. + + Once the database can be analysed, Ora2Pg, by his ability to convert SQL + and PL/SQL code from Oracle syntax to PostgreSQL, can go further by + estimating the code difficulties and estimate the time necessary to + operate a full database migration. + + To estimate the migration cost in man-days, Ora2Pg allow you to use a + configuration directive called ESTIMATE_COST that you can also enabled + at command line: + + --estimate_cost + + This feature can only be used with the SHOW_REPORT, FUNCTION, PROCEDURE, + PACKAGE and QUERY export type. + + ora2pg -t SHOW_REPORT --estimate_cost + + The generated report is same as above but with a new 'Estimated cost' + column as follow: + + -------------------------------------- + Ora2Pg: Oracle Database Content Report + -------------------------------------- + Version Oracle Database 10g Express Edition Release 10.2.0.1.0 + Schema HR + Size 890.00 MB + + -------------------------------------- + Object Number Invalid Estimated cost Comments + -------------------------------------- + DATABASE LINK 3 0 9 Database links will be exported as SQL/MED PostgreSQL's Foreign Data Wrapper (FDW) extensions + using oracle_fdw. + FUNCTION 2 0 7 Total size of function code: 369 bytes. HIGH_SALARY: 2, VALIDATE_SSN: 3. + INDEX 21 0 11 11 index(es) are concerned by the export, others are automatically generated and will do so + on PostgreSQL. 11 b-tree index(es). Note that bitmap index(es) will be exported as b-tree + index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. + Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a + reverse() function based index and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' + or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator + respectively into varchar, text or char columns. + JOB 0 0 0 Job are not exported. You may set external cron job with them. + MATERIALIZED VIEW 1 0 3 All materialized view will be exported as snapshot materialized views, they + are only updated when fully refreshed. + PACKAGE BODY 0 2 54 Total size of package code: 2487 bytes. Number of procedures and functions found + inside those packages: 7. two_proc.get_table: 10, emp_mgmt.create_dept: 4, + emp_mgmt.hire: 13, emp_mgmt.increase_comm: 4, emp_mgmt.increase_sal: 4, + emp_mgmt.remove_dept: 3, emp_mgmt.remove_emp: 2. + PROCEDURE 4 0 39 Total size of procedure code: 2436 bytes. TEST_COMMENTAIRE: 2, SECURE_DML: 3, + PHD_GET_TABLE: 24, ADD_JOB_HISTORY: 6. + SEQUENCE 3 0 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL + will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). + SYNONYM 3 0 4 SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround + is to use views or set the PostgreSQL search_path in your session to access + object outside the current schema. + user1.emp_details_view_v is an alias to hr.emp_details_view. + user1.emp_table is an alias to hr.employees@other_server. + user1.offices is an alias to hr.locations. + TABLE 17 0 8.5 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration + directive to export as file_fdw foreign tables or use COPY in your code if you just want to + load data from external files. 2 binary columns. 4 unknown types. + TRIGGER 1 1 4 Total size of trigger code: 123 bytes. UPDATE_JOB_HISTORY: 2. + TYPE 7 1 5 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. + 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are + converted as table, type inheritance is not supported. + TYPE BODY 0 3 30 Export of type with member method are not supported, they will not be exported. + VIEW 1 1 1 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. + -------------------------------------- + Total 65 8 162.5 162.5 cost migration units means approximatively 2 man day(s). + + The last line shows the total estimated migration code in man-days + following the number of migration units estimated for each object. This + migration unit represent around five minutes for a PostgreSQL expert. If + this is your first migration you can get it higher with the + configuration directive COST_UNIT_VALUE or the --cost_unit_value command + line option: + + ora2pg -t SHOW_REPORT --estimate_cost --cost_unit_value 10 + + Ora2Pg is also able to give you a migration difficulty level assessment, + here a sample: + + Migration level: B-5 + + Migration levels: + A - Migration that might be run automatically + B - Migration with code rewrite and a human-days cost up to 5 days + C - Migration with code rewrite and a human-days cost above 5 days + Technical levels: + 1 = trivial: no stored functions and no triggers + 2 = easy: no stored functions but with triggers, no manual rewriting + 3 = simple: stored functions and/or triggers, no manual rewriting + 4 = manual: no stored functions but with triggers or views with code rewriting + 5 = difficult: stored functions and/or triggers with code rewriting + + This assessment consist in a letter A or B to specify if the migration + needs manual rewriting or not. And a number from 1 up to 5 to give you a + technical difficulty level. You have an additional option + --human_days_limit to specify the number of human-days limit where the + migration level should be set to C to indicate that it need a huge + amount of work and a full project management with migration support. + Default is 10 human-days. You can use the configuration directive + HUMAN_DAYS_LIMIT to change this default value permanently. + + This feature has been developed to help you or your boss to decide which + database to migrate first and the team that must be mobilized to operate + the migration. + + Global Oracle and MySQL migration assessment + Ora2Pg come with a script ora2pg_scanner that can be used when you have + a huge number of instances and schema to scan for migration assessment. + + Usage: ora2pg_scanner -l CSVFILE [-o OUTDIR] + + -b | --binpath DIR: full path to directory where the ora2pg binary stays. + Might be useful only on Windows OS. + -c | --config FILE: set custom configuration file to use otherwise ora2pg + will use the default: /etc/ora2pg/ora2pg.conf. + -l | --list FILE : CSV file containing a list of databases to scan with + all required information. The first line of the file + can contain the following header that describes the + format that must be used: + + "type","schema/database","dsn","user","password" + + -o | --outdir DIR : (optional) by default all reports will be dumped to a + directory named 'output', it will be created automatically. + If you want to change the name of this directory, set the name + at second argument. + + -t | --test : just try all connections by retrieving the required schema + or database name. Useful to validate your CSV list file. + -u | --unit MIN : redefine globally the migration cost unit value in minutes. + Default is taken from the ora2pg.conf (default 5 minutes). + + Here is a full example of a CSV databases list file: + + "type","schema/database","dsn","user","password" + "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" + "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" + + The CSV field separator must be a comma. + + Note that if you want to scan all schemas from an Oracle instance you just + have to leave the schema field empty, Ora2Pg will automatically detect all + available schemas and generate a report for each one. Of course you need to + use a connection user with enough privileges to be able to scan all schemas. + For example: + + "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" + + will generate a report for all schema in the XE instance. Note that in this + case the SCHEMA directive in ora2pg.conf must not be set. + + It will generate a CSV file with the assessment result, one line per + schema or database and a detailed HTML report for each database scanned. + + Hint: Use the -t | --test option before to test all your connections in + your CSV file. + + For Windows users you must use the -b command line option to set the + directory where ora2pg_scanner stays otherwise the ora2pg command calls + will fail. + + In the migration assessment details about functions Ora2Pg always + include per default 2 migration units for TEST and 1 unit for SIZE per + 1000 characters in the code. This mean that by default it will add 15 + minutes in the migration assessment per function. Obviously if you have + unitary tests or very simple functions this will not represent the real + migration time. + + Migration assessment method + Migration unit scores given to each type of Oracle database object are + defined in the Perl library lib/Ora2Pg/PLSQL.pm in the %OBJECT_SCORE + variable definition. + + The number of PL/SQL lines associated to a migration unit is also + defined in this file in the $SIZE_SCORE variable value. + + The number of migration units associated to each PL/SQL code + difficulties can be found in the same Perl library lib/Ora2Pg/PLSQL.pm + in the hash %UNCOVERED_SCORE initialization. + + This assessment method is a work in progress so I'm expecting feedbacks + on migration experiences to polish the scores/units attributed in those + variables. + + Improving indexes and constraints creation speed + Using the LOAD export type and a file containing SQL orders to perform, + it is possible to dispatch those orders over multiple PostgreSQL + connections. To be able to use this feature, the PG_DSN, PG_USER and + PG_PWD must be set. Then: + + ora2pg -t LOAD -c config/ora2pg.conf -i schema/tables/INDEXES_table.sql -j 4 + + will dispatch indexes creation over 4 simultaneous PostgreSQL + connections. + + This will considerably accelerate this part of the migration process + with huge data size. + + Exporting LONG RAW + If you still have columns defined as LONG RAW, Ora2Pg will not be able + to export these kind of data. The OCI library fail to export them and + always return the same first record. To be able to export the data you + need to transform the field as BLOB by creating a temporary table before + migrating data. For example, the Oracle table: + + SQL> DESC TEST_LONGRAW + Name NULL ? Type + -------------------- -------- ---------------------------- + ID NUMBER + C1 LONG RAW + + need to be "translated" into a table using BLOB as follow: + + CREATE TABLE test_blob (id NUMBER, c1 BLOB); + + And then copy the data with the following INSERT query: + + INSERT INTO test_blob SELECT id, to_lob(c1) FROM test_longraw; + + Then you just have to exclude the original table from the export (see + EXCLUDE directive) and to renamed the new temporary table on the fly + using the REPLACE_TABLES configuration directive. + + Global variables + Oracle allow the use of global variables defined in packages. Ora2Pg + will export these variables for PostgreSQL as user defined custom + variables available in a session. Oracle variables assignment are + exported as call to: + + PERFORM set_config('pkgname.varname', value, false); + + Use of these variables in the code is replaced by: + + current_setting('pkgname.varname')::global_variables_type; + + where global_variables_type is the type of the variable extracted from + the package definition. + + If the variable is a constant or have a default value assigned at + declaration, Ora2Pg will create a file global_variables.conf with the + definition to include in the postgresql.conf file so that their values + will already be set at database connection. Note that the value can + always modified by the user so you can not have exactly a constant. + + Hints + Converting your queries with Oracle style outer join (+) syntax to ANSI + standard SQL at the Oracle side can save you lot of time for the + migration. You can use TOAD Query Builder can re-write these using the + proper ANSI syntax, see: + http://www.toadworld.com/products/toad-for-oracle/f/10/t/9518.aspx + + There's also an alternative with SQL Developer Data Modeler, see + http://www.thatjeffsmith.com/archive/2012/01/sql-developer-data-modeler- + quick-tip-use-oracle-join-syntax-or-ansi/ + + Toad is also able to rewrite the native Oracle DECODE() syntax into ANSI + standard SQL CASE statement. You can find some slide about this in a + presentation given at PgConf.RU: + http://ora2pg.darold.net/slides/ora2pg_the_hard_way.pdf + + Test the migration + The type of action called TEST allow you to check that all objects from + Oracle database have been created under PostgreSQL. Of course PG_DSN + must be set to be able to check PostgreSQL side. + + Note that this feature respect the schema set in the SCHEMA directive to + scan the Oracle database and also at PostgreSQL side if EXPORT_SCHEMA is + enabled. If PG_SCHEMA is defined and EXPORT_SCHEMA is enabled Ora2Pg + will use the list of schemas defined in PG_SCHEMA to scan PostgreSQL. If + EXPORT_SCHEMA is disabled the entire PostgreSQL database is scanned. + + For example command: + + ora2pg -t TEST -c config/ora2pg.conf > migration_diff.txt + + Will create a file containing the report of all object and row count on + both side, Oracle and PostgreSQL, with an error section giving you the + detail of the differences for each kind of object. Here is a sample + result: + + [TEST ROWS COUNT] + ORACLEDB:COUNTRIES:25 + POSTGRES:countries:25 + ORACLEDB:CUSTOMERS:6 + POSTGRES:customers:6 + ORACLEDB:DEPARTMENTS:27 + POSTGRES:departments:27 + ORACLEDB:EMPLOYEES:107 + POSTGRES:employees:107 + ORACLEDB:JOBS:19 + POSTGRES:jobs:19 + ORACLEDB:JOB_HISTORY:10 + POSTGRES:job_history:10 + ORACLEDB:LOCATIONS:23 + POSTGRES:locations:23 + ORACLEDB:PRODUCTS:0 + POSTGRES:products:0 + ORACLEDB:PTAB2:4 + ORACLEDB:REGIONS:4 + POSTGRES:regions:4 + [ERRORS ROWS COUNT] + Table ptab2 does not exists in PostgreSQL database. + + [TEST INDEXES COUNT] + ORACLEDB:COUNTRIES:1 + POSTGRES:countries:1 + ORACLEDB:JOB_HISTORY:4 + POSTGRES:job_history:4 + ORACLEDB:DEPARTMENTS:2 + POSTGRES:departments:1 + ORACLEDB:EMPLOYEES:6 + POSTGRES:employees:6 + ORACLEDB:CUSTOMERS:1 + POSTGRES:customers:1 + ORACLEDB:REGIONS:1 + POSTGRES:regions:1 + ORACLEDB:LOCATIONS:4 + POSTGRES:locations:4 + ORACLEDB:JOBS:1 + POSTGRES:jobs:1 + [ERRORS INDEXES COUNT] + Table departments doesn't have the same number of indexes in Oracle (2) and in PostgreSQL (1). + + [TEST VIEW COUNT] + ORACLEDB:VIEW:1 + POSTGRES:VIEW:1 + [ERRORS VIEW COUNT] + OK, Oracle and PostgreSQL have the same number of VIEW. + + [TEST MVIEW COUNT] + ORACLEDB:MVIEW:0 + POSTGRES:MVIEW:0 + [ERRORS MVIEW COUNT] + OK, Oracle and PostgreSQL have the same number of MVIEW. + + [TEST SEQUENCE COUNT] + ORACLEDB:SEQUENCE:1 + POSTGRES:SEQUENCE:0 + [ERRORS SEQUENCE COUNT] + SEQUENCE does not have the same count in Oracle (1) and in PostgreSQL (0). + + [TEST TYPE COUNT] + ORACLEDB:TYPE:1 + POSTGRES:TYPE:0 + [ERRORS TYPE COUNT] + TYPE does not have the same count in Oracle (1) and in PostgreSQL (0). + + [TEST FDW COUNT] + ORACLEDB:FDW:0 + POSTGRES:FDW:0 + [ERRORS FDW COUNT] + OK, Oracle and PostgreSQL have the same number of FDW. + + Here we can see that one table, one index, one sequence and one user + defined type have not been imported yet or have encountered an error. + +SUPPORT + Author / Maintainer + Gilles Darold + + Please report any bugs, patches, help, etc. to . + + Feature request + If you need new features let me know at . This + help a lot to develop a better/useful tool. + + How to contribute ? + Any contribution to build a better tool is welcome, you just have to + send me your ideas, features request or patches and there will be + applied. + +LICENSE + Copyright (c) 2000-2020 Gilles Darold - All rights reserved. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see < http://www.gnu.org/licenses/ >. + +ACKNOWLEDGEMENT + I must thanks a lot all the great contributors, see changelog for all + acknowledgments. + diff --git a/README.md b/README.md index 08317416b9fc7bdb2a639ef88d5d9b2597bc67d0..2b2364d5e33fc0a0afbf0de9313bd3ccadd115c0 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,63 @@ # openGauss-tools-ora2og #### 介绍 -{**以下是 Gitee 平台说明，您可以替换此简介** -Gitee 是 OSCHINA 推出的基于 Git 的代码托管平台（同时支持 SVN）。专为开发者提供稳定、高效、安全的云端软件开发协作平台 -无论是个人、团队、或是企业，都能够用 Gitee 实现代码托管、项目管理、协作开发。企业项目请看 [https://gitee.com/enterprises](https://gitee.com/enterprises)} +ora2og是一个将Oracle数据库迁移至openGauss的工具，主要编程语言为perl，通过perl DBI模块连接Oracle数据库，自动扫描并提取其中的对象结构及数据，产生SQL脚本，通过手动或自动的方式应用到openGauss。此外，工具还提供丰富配置项，用户可以自定义迁移行为。 -#### 软件架构 -软件架构说明 +ora2og初始代码源自ora2pg，版本为release v21.1：https://github.com/darold/ora2pg/tree/v21.1。 + +#### 优秀特性 +* 支持导出数据库绝大多数对象类型，包括表、视图、序列、索引、外键、约束、函数、存储过程等。 + +* 提供PL/SQL到PL/PGSQL语法的自动转换，一定程度避免了人工修正。 + +* 可生成迁移报告，包括迁移难度评估、人天估算。 + +* 可选对导出数据进行压缩，节约磁盘开销。 + +* 配置选项丰富，可自定义迁移行为。 #### 安装教程 -1. xxxx -2. xxxx -3. xxxx +1. 安装依赖 +由于编程语言为perl，需要安装所需perl模块。此外，还需要DBI、DBD::Pg、DBD::Oracle连接源和目标数据库。请在root用户下执行。 +```shell +yum install -y perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker +yum install perl-CPAN +perl -MCPAN -e 'install DBI' +perl -MCPAN -e 'install DBD::Pg' +``` +安装DBD::Oracle，需要先安装Oracle Instant Client或者本地已安装Oracle数据库。 +```shell +# 从Oracle官方下载并安装Oracle Instant Client(x86版本,ARM环境下请下载对应的RPM包) +rpm -ivh oracle-instantclient19.11-basic-19.11.0.0.0-1.x86_64.rpm +rpm -ivh oracle-instantclient19.11-devel-19.11.0.0.0-1.x86_64.rpm +rpm -ivh oracle-instantclient19.11-jdbc-19.11.0.0.0-1.x86_64.rpm +rpm -ivh oracle-instantclient19.11-sqlplus-19.11.0.0.0-1.x86_64.rpm +# 设置环境变量ORACLE_HOME +export ORACLE_HOME=/usr/lib/oracle/19.11/client64/ +# 或者本地已安装有Oracle数据库 +ORACLE_HOME如下设置 +export ORACLE_HOME=/opt/oracle/product/19c/dbhome_1 +export LD_LIBRARY_PATH=$ORACLE_HOME/lib +# 安装DBD:Oracle +perl -MCPAN -e 'install DBD::Oracle' +``` +2. 安装Ora2Pg +为目标安装路径，为下载的代码路径。 +```shell +# 进到代码目录下 +perl Makefile.PL PREFIX= +make && make install +# 设置环境变量，查看是否安装成功 +export PERL5LIB=/lib +export PATH=$PATH:/usr/local/bin +ora2pg --help +``` #### 使用说明 -1. xxxx -2. xxxx -3. xxxx +1. 如该文章所示：https://mp.weixin.qq.com/s/hMqaSes0hQvzmJw0kmXDtg #### 参与贡献 @@ -27,13 +65,3 @@ Gitee 是 OSCHINA 推出的基于 Git 的代码托管平台（同时支持 SVN 2. 新建 Feat_xxx 分支 3. 提交代码 4. 新建 Pull Request - - -#### 特技 - -1. 使用 Readme\_XXX.md 来支持不同的语言，例如 Readme\_en.md, Readme\_zh.md -2. Gitee 官方博客 [blog.gitee.com](https://blog.gitee.com) -3. 你可以 [https://gitee.com/explore](https://gitee.com/explore) 这个地址来了解 Gitee 上的优秀开源项目 -4. [GVP](https://gitee.com/gvp) 全称是 Gitee 最有价值开源项目，是综合评定出的优秀开源项目 -5. Gitee 官方提供的使用手册 [https://gitee.com/help](https://gitee.com/help) -6. Gitee 封面人物是一档用来展示 Gitee 会员风采的栏目 [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/) diff --git a/changelog b/changelog new file mode 100644 index 0000000000000000000000000000000000000000..2ae44f2dcdadd20a062656479dec33508877db4a --- /dev/null +++ b/changelog @@ -0,0 +1,5676 @@ +2021 04 01 - v21.1 + +This release fix several issues reported since past six months and +as usual adds some new features and improvements. + + * Now that Orafce 3.15.0 has a definition for the REGEXP_* function, + makes the translation optional to USE_ORAFCE directive. + * Add set application name in connection to Oracle/MySql/PostgreSQL. + * Add translation of REGEXP_COUNT() and change assessment cost. + * Rewrite the way REGEXP_LIKE() is translated into regexp_match to + support modifiers. This rewrite also fix default behavior between + Oracle and PostgreSQL. + * Replace DBMS_LOB.GETLENGTH() by PostgreSQL octet_length() function. + * Add types correspondences for VARCHAR2 and NVARCHAR2 in DATA_TYPE + configuration directive. + * Add autodetection and support of geometry type, srid and dimension + for ArcGis geometries. + * Add conversion of default value in function parameters. + * Add -u | --unit option to ora2pg_scanner to be able to set the + migration cost unit value globally. + * Replace DBMS_LOB.SUBSTR() by SUBSTR() + * Remove TO_CLOB() it is useless, manual cast could be necessary. + * Replace IS JSON validation clause in CHECK constraints by + (CASE WHEN $1::json IS NULL THEN true ELSE true END) + When the code is invalid an error is fired. + * DISTINCT and UNIQUE are synonym on Oracle + +Backward compatibility changes: + + - Force a column to be bigint if this is an identity column. Thanks + to MigOps.com for the patch. + - Fix EMPTY_LOB_NULL, enable/disable was inverted, keeping default + to enabled. Take care that in old ora2pg.conf it is disabled so it + will break backward compatibility with old configuration. + - Replace NO_LOB_LOCATOR with USE_LOB_LOCATOR and NO_BLOB_EXPORT + with ENABLE_BLOB_EXPORT to avoid confusion with double negative + variable. Backward compatibility is preserved with a warning. + - SRID for SDO_GEOMETRY export is now taken from the value not forced + from the metadata table. + +Here is the full list of changes and acknowledgements: + + - Take Geometry SRID from the data and fallback to SRID defined in + metadata when not found. Thanks to Sebastian Albert for the report. + - Fix case where Ora2Pg temporary substitution of '' by placeholder + was not restored. Thanks to MigOps.com for the patch. + - Fix identity column export on unsupported Oracle 18c options. + Thanks to MigOps.com for the patch. + - Fix export of columns indexes created with single quote. + Thanks to MigOps.com for the patch. + - Fix replacement of keyword PROCEDURE by FUNCTION in constraints + constants definition. Thanks to marie-joechahine for the report. + - Replace IS JSON validation clause in CHECK constraints. Thanks to + marie-joechahine for the report and MigOps.com for the patch. + - Add support to ON OVERFLOW clause in LISTAGG replacement. + Thanks to MigOps.com for the patch. + - Fix incorrect handling of HAVING+GROUP BY rewriting. + Thanks to MigOps.com for the patch. + - Add replacement of TO_NCHAR by a cast to varchar. Thanks to + MigOps.com for the patch. + - Fix replacement of NOTFOUND when there is extra space or new line + in the WHEN EXIT clause. Thanks to MigOps.com for the patch. + - Fix a regression in NO_VIEW_ORDERING, it was not taken in account + anymore. Thanks to RonJojn2 for the report. + - Replace DATA_TYPE with DTD_IDENTIFIER in MySQL catalog queries for + version prior 5.5.0. Thanks to zejeanmi for the report. + - Fix import script to import sequences before tables. Thanks to + MigOps.com for the patch. + - Fix detail report of custom type in migration assessment. Thanks + to MigOps.com for the patch. + - Fix duplicate schema prefixed to SYNONYM. Thanks to dlc75 for the + reports. + - Replace NO_LOB_LOCATOR with USE_LOB_LOCATOR and NO_BLOB_EXPORT with + ENABLE_BLOB_EXPORT to avoid confusion with double negative variable. + Thanks to Rob Johnson for the report. + - Fix some missing replacements of NVL and rewrite !=-1 into != -1. + Thanks to MigOps.com for the patch. + - Fix ROWNUM followed by + or - operator and when no aliases are + provided. Thanks to MigOps.com for the patch. + - Add DBSFWUSER to the list of user/schema exclusion. Thanks to + MigOps.com for the patch. + - Fix regexp to not append subquery aliases on JOIN clause. Thanks + to Rui Pereira for the report. + - Handle PRESERVE_CASE and EXPORT_SCHEMA in sequence name. Thanks + to marie-joechahine for the report. + - Add CREATE SCHEMA statement to sequence export when EXPORT_SCHEMA + is enabled. Thanks to marie-joechahine for the report. + - Fix duplicate index name on subpartition. Thanks to Philippe + Beaudoin for the report. + - Exclude sequences used for IDENTITY column (ISEQ$$_). Thanks to + marie-joechahine for the report. + - Fix parsing from file of CREATE SEQUENCE. Thanks to Rui Pereira + for the report. + - In export_all.sh script use the database owner provided if it is a + superuser instead of postgres user. Thanks to jjune235 for the + feature request. + - Fix parsing of triggers when there is a CASE inside the code. + Thanks to Rui Pereira for the report. + - Add set application name in connection to Oracle/MySql/PostgreSQL. + Thanks to Yoni Sade for the patch. + - Fix double column alias when replacing ROWNUM. Thanks to Rui + Pereira for the report. + - Add translation of the REGEXP_COUNT function and change assessment + cost. + - Rewrite the way REGEXP_LIKE is translated into regexp_match to + support modifiers. This rewrite also fix default behavior between + Oracle and PostgreSQL. Thanks to otterrisk for the report. + - Add IS JSON to assessment. Thanks to marie-joe Chahine for the + report. + - Fix multi-columns RANGE partitioning. Thanks to Philippe Beaudoin + for the report. + - Improve reordering columns. Sort by fieldsize first, if same size + then it sorts by original position. Thanks to Sebastien Caunes for + the patch. + - Append partition's column to the primary key of the table as it + must be part of the PK on PostgreSQL. Thanks to xinjirufen for the + report. + - Fix partition export where PRESERVE_CASE was applied to Oracle + side. Thanks to schleb1309 for the report. + - Fix trigger export with column restriction. Thanks to Sebastien + Caunes for the report. + - Update installation information. + - Fix table reordering following data type. Thanks to Sebastien + Caunes for the patch. + - Fix incorrect variable name corresponding to DATA_EXPORT_ORDER + making this directive inefficient. Thanks to Ron Johnson for the + report. + - Fix translation of check constraint when read from file + - Fix EMPTY_LOB_NULL, enable/disable as inverted, keep default to + enabled. Take care that in old ora2pg.conf it is disabled so it + will break backward compatibility with old configuration. + - Fix false positive detection of input filename is the same as + output file. + - Rename variables SCHEMA_ONLY, DATA_ONLY and CONSTRAINTS_ONLY in + script import_all.sh to conform to their real use. Thanks to + Sebastien Caunes for the report. + - Fix comment detection breaking the package header parsing and + global variable detection. + - Fix ROWNUM detection for replacement by LIMIT + - Fix escaping of psql command in configuration file comment and + set default value for PG_VERSION to 12. + - Replace precision by exactness in documentation. Thanks to + Sebastien Caunes for the report. + - Prevent reducing DATA_LIMIT when NO_BLOB_EXPORT is enabled. + Thanks to Thomas Reiss for the report. + - Fix geometry type detection. + - Add autodetection of geometry type, srid and dimension for + ArcGis geometries. Thanks to changmao01 for the feature request. + - Fix call to ST_GeomFromText when no SRID is found. + - Fix case where OVERRIDE SYSTEM VALUE clause could be added if PG + version is < 10. Thanks to changmao01 for the report. + - Fix unwanted call to internal GEOM library for ArcGis geometries. + Thanks to changmao01 for the report. + - Exclude schema SDE (ArGis) from export. Thanks to changmao01 for + the report. + - prevent looking twice to same custom data type definition. + - Fix previous patch to catch SDO_GEOMETRY on lowercase regexp. + - Limit detection of geometry data type to SDO_GEOMETRY. + - Fix column name replacement in view definition. Thanks to Amit + Sanghvi for the report. + - Fix REPLACE_COLS parsing to allow space in column name. Thanks + to Amit Sanghvi for the report. + - Fix translation from file of triggers with WHEN clause. Thanks + to Rui Pereira for the report. + - Fix column name kept lowercase in the MOD() clause when -J is + used. Thanks to Code-UV for the report. + - Keep case of PG_SCHEMA definition when used in TEST action. + - Fix data export for columns with custom data type. Thanks to + Aymen Zaiter for the report. + - Fix missing bracket with || operator in CREATE INDEX. Thanks to + traxverlis for the report. + - Fix export of single row unique function base index. Example: + CREATE UNIQUE INDEX single_row_idx ON single_row ((1)); + Thanks to unrandom123 for the report. + - Update documentation about schemas used in TEST action. + - Disable materialized view export with MySQL export it is not + supported. Thanks to naveenjul29 for the report. + - Fix table alias detection in Oracle (+) join rewrite. + - Fix an infinite loop in Oracle (+) join rewrite when there is no + table aliases and the table is prefixed by its schema. Thanks to + Olivier Picavet for the report. + - Fix MODIFY_STRUCT when column name need to be escaped. Thanks to + helmichamsi10 for the report. + - Fix empty PARTITION BY () clause. Thanks to Aymen Zaiter. + - Fix export of global variable from package description when there + is no package body. Thanks to naveenjul29 for the report. + - Add package description export when dumping package source, + previously only the package body was dump. This will allow to + check global variables export. + - Whilst working on the Reproducible Builds effort (https//reproducible-builds.org/) + it appears that ora2pg could not be built reproducibly. Thanks to + Chris Lamb for the patch. + - Fix case of NUMBER(*,10) declaration. Oracle has a precision of 1 to 38 + for numeric. Even if PostgreSQL allow a precision of 1000 use 38 to + replace junk parameter. Thanks to xinjirufen for the report. + - Add conversion of default value in function parameters, like syssdate + rewriting for example. Thanks to unrandom123 for the report. + - Fix a regression in data encoding when exporting data introduced in + commit fa8e9de. Thanks to gp4git for the report. + - Add debug information about the environment variables used before + connecting to Oracle. + - Fix case of duplicate between unique index and unique constraint with + multiple columns. Thanks to gp4git. + +2020 10 12 - v21.0 + +This release fix several issues reported since last release and adds +several new features and improvements. + + * Add clause OVERRIDING SYSTEM VALUE to INSERT statements when the + table has an IDENTITY column. + * Considerably increase the speed to generate the report about the + migration assessment, especially for database with huge number of + objects. + * Reduce time passed in the progress bar. Following the number of + database objects we were spending too much time in refreshing the + progress bar. + * Add number of identity columns in migration assessment report. + * Make assessment details report initially hidden using HTML5 tags +
+ * Improve speed of BLOB/CLOB data export. Oracle recommends reading + from and writing to a LOB in batches using a multiple of the LOB + chunk size. This chunk size defaults to 8k (8192). Recent tests + show that the best performances can be reach with higher value + like 512K or 4Mb. + * Add progress bar when --oracle_speed is use in single process mode. + * Automatically activate USER_GRANTS when the connection user has no DBA + privilege. A warning is displayed. + * Complete port to Windows by using the Windows separator on stdout + redirection into a file at ora2pg command line call and improve + ora2pg_scanner port on Windows OS. + * Add rewrite of MySQL JOIN with WHERE clause instead of ON. + * Add MGDSYS (Oracle E-Business Suite) and APEX_040000 to the list + of schemas excluded from the export. + * Supply credentials interactively when a password is not defined in + the configuration file. Need the installation of a new Perl module + Term::ReadKey. + * Add supports oracle connections "as sysdba" with username "/" and + an empty password to connect to a local oracle instance. + * Add translation of PRIVATE TEMPORARY TABLE from Oracle 18c into + PostgreSQL basic temporary table, only the default behavior for + on commit change. + +New command line options: + + * Add new command line option to ora2pg_scanner: -b | --binpath DIR + to set the full path to directory where the ora2pg binary stays. + Might be useful only on Windows OS. + * Add -r | --relative command line option and PSQL_RELATIVE_PATH + configuration directive. By default Ora2Pg use \i psql command to + execute generated SQL files if you want to use a relative path + following the script execution file enabling this option will use + \ir. See psql help for more information. + +New configuration directives: + + * NO_VIEW_ORDERING: + By default Ora2Pg try to order views to avoid error at import time + with nested views. With a huge number of views this can take a very + long time, you can bypass this ordering by enabling this directive. + * NO_FUNCTION_METADATA + Force Ora2Pg to not look for function declaration. Note that this + will prevent Ora2Pg to rewrite function replacement call if needed. + Do not enable it unless looking forward at function breaks other + export. + * LOB_CHUNK_SIZE + See explanation in the new features and improvement list. + * ALTERNATIVE_QUOTING_REGEXP + To support the Alternative Quoting Mechanism ('Q' or 'q') for String + Literals set the regexp with the text capture to use to extract the + text part. For example with a variable declared as + c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; + the regexp to use must be: + ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' + ora2pg will use the $$ delimiter, with the example the result will + be: + c_sample varchar(100) := $$This doesn't work.$$; + The value of this configuration directive can be a list of regexp + separated by a semi colon. The capture part (between parenthesis) is + mandatory in each regexp if you want to restore the string constant. + +Backward compatibility changes: + + - Default for NO_LOB_LOCATOR is now 1 to benefit from the LOB_CHUNK_SIZE + performances gain. + - Enable schema compilation (COMPILE_SCHEMA set to 1) by default to + speed up DDL extraction. + - Change the behavior of Ora2Pg with the parameters that follows a + parameter with a default value. Ora2Pg used to change the order of the + parameter's function to put all parameters with a default value at end + of the list which need a function call rewrite. This have been abandoned + now any parameter without default value after a parameter with a default + value will be appended DEFAULT NULL. + +Here is the full list of changes and acknowledgements: + + - Fix unwanted references to PK/UK when DROP_INDEXES is enabled. + - Fix comparison between function name in TEST report. + - Fix duplicates on retrieving partitions information. + - Improve SHOW_TABLE report about partitioned tables information. + - Drop code about removing DEFAULT NULL in functions parameters. Thanks to + chaluvadi286 for the report. + - Fix two other case where materialized view can be listed in the table list. + - Fix case where materialized view can be listed in the table list. Thanks + to Thomas Reiss for the report. + - Fix %ROWTYPE removing to be restricted to REF CURSOR. Thanks to + jagmohankaintura-tl for the report. + - Fix PG functions count when comparing Oracle functions count in TEST action. + Remove useless -l option to import_all.sh auto generated script. + - Fix PRESERVE_CASE on schema name for functions extracted from a package. + - Fix search_path adding public default schema. + - Apply PRESERVE_CASE to partition by involved columns. + - Add IF EXIXTS to create schema to avoid error when import_all.sh is run + several time. + - Fix sort order of comment on columns for tables and views. + - Fix warning about data export from nonexistent table resulting of index + lookup on nested table. + - Fix infinite loop in global variables package extraction. Thanks to Thomas + Reiss for the report. + - Fix global variables and packages export when comments are present in the + package description. + - Add information about XML_PRETTY size limit to 4000 + - Fix column name in indexes when PRESERVE_CASE is enabled. Thanks + to Julien traxverlis for the report. + - Fix Top 10 of largest tables sort order. Thanks to Tom Vanzieleghem + for the patch. + - Fix duplicates between indexes and constraints. Thanks to sdpdb and + Jon Betts for the report. + - Fix SYSDATE replacement and possible infinite loop in SYSDATE parsing. + Thanks to pbidault for the report. + - Fix export of Oracle TEXT indexes with USE_UNACCENT disabled. Thanks to + Eric Delanoe for the report. + - Add new configuration directive ALTERNATIVE_QUOTING_REGEXP to support + the Alternative Quoting Mechanism ('Q' or 'q') for String Literals. + Thanks to just-doit for the report. + - Fix OF clause missing in update triggers. Thanks to just-doit for + the report. + - Fix IS NULL translation in WHERE clause of UPDATE statement. Thanks + to Eric Delanoe for the report. + - Remove DDL export of LOG indexes on materialized views. + - Fix unexpected materialized view listed in table export. Thanks to + jagmohankaintura-tl for the report. + - Fix default values with single quote in create table DDL. Thanks to + justdoit for the report. + - Fix double quote in CREATE TRIGGER code and applying of preserve case + on column name. + - Supply credentials interactively when a password is not defined in + configuration file. Thanks to rpeiremans for the patch. + - Add supports oracle connections "as sysdba" with username "/" and + an empty password to connect to a local oracle instance. Thanks to + rpeiremans for the patch. + - Fix documentation about materialized view export. + - Fix export order of comments on columns. + - Fix export of views comments when no schema is used for export and + export schema is activated. + - Fix cast in replacement with TO_NUMBER and TO_CHAR in indexes. Thanks + to Kiran for the report. + - Add MGDSYS (Oracle E-Business Suite) to the list of schemas excluded + from the export. Thanks to naveenjul29 for the report. + - Add more information about PG_DSN use. Thanks to Pepan7 for the report. + - Update copyright year. + - Fix regression where "SET client_encoding TO ..." was missing data file + header. Thanks to Emmanuel Gaultier for the report. + - Fix EDITABLE vs EDITIONABLE parsing. Thanks to Naveen Kumar for the report. + - Fix typos in documentation. Thanks to swallow-life, ChrisYuan, Edward Betts, + Jack Caperon and cavpollo for the patches. + - Add OVERRIDING SYSTEM VALUE to INSERT statement when the table has an + IDENTITY column. Thanks to Robin Windey for the report + - Remove empty parenthesis of identity column options + - Limit sequence/identity column value to bigint max + - Add an example of DBD::Oracle DSN with 18c. + - Fix parsing of identity column from file. Thanks to deepakp555 for the + report. + - Fix quoting of identifier when PRESERVE_CASE is enable and no particular + schema is specified. Thanks to mkgrgis for the report. + - Move setting of search_path before truncate table. Thanks to Michael Vitale + for the report. + - Add explanation about TEST and SIZE migration assessment values. + - Mark XMLTYPE as having LOB locator. + - Fix XMLTYPE columns that are exported as lob locator. Thanks to Tamas for + the report. + - Fix a problem of data export throughput that was slowing down all along + the export when multiprocess for output was not used. Ora2Pg was forking + a process for each chunk of data (see DATA_LIMIT) which is useless when + write output is done on a single process (-j 1) and slow down the export. + Thanks to markhooper99 and Tamas for reporting, testing and finding the + source of the issue. + - Fix progress bar in multiprocess mode, update was not displayed at each + chunk of data processed. + - Add internal debug information for progress bar. + - Add debug information for SHOW_REPORT + - Fix a long pending issue with custom data type export. Thanks to + jhollandsworth for the patch. + - Fix LOB data export with value changed to NULL when the CLOB value was 0. + Thanks to jhollandsworth for the report. + - Fix escape format issue with COPY and bytea. Thanks to Christoph Noel and + dwbrock62 for the report. + - Add LD_LIBRARY_PATH and PATH prerequisite to run ora2pg. + - Fix use of the HIGH_VALUE column in partition listing with Oracle 9i. Thanks + to Francisco Puga for the report. + - Update the table row count logic to incorporate the PostgreSQL table FQN as + established through the set_pg_relation_name routine. Thanks to Jacob + Roberts for the patch. + - Add the PostgreSQL FQN when printing the results in the TEST function. Thanks + to Jacob Roberts for the patch. + - Do not look forward function with the SHOW_* action + - Fix BLOB export where \x was escaped. Thanks to Christophe Noel for the + report. + - Update Ora2Pg.pm to fix symbol in column name in create index statement. + Thanks to kpoluektov for the patch. + - Fix package function extraction when there is a start of comment (/*) in + a constant string. Thanks to Tiago Anastacio for the report. + - Fix type detection in package declaration. Thanks to Tiago Anastacio for + the report. + - Avoid displaying error ORA-22831 when exporting LOB. This error can + appears when LOB chunk size is different from default 8192. The error + has no incidence on the export so we can just ignore it. This patch + also use DBD::Oracle ora_lob_chunk_size() method to gather chunk the + chunk size of the LOB, fallback to 8192 if not available. Thanks to + joedbadmin for the report. + - Disable direct report of Oracle errors, all error should be handled at + Ora2Pg level. + - Fix MySQL data export with allow/exclude objects. Thanks to Manuel Pavy for + the report. + - Fix exclude/allow object feature in MySQL export that was not working since + release 19.0. Thanks to Manuel Pavy for the report. + - Add rewrite of MySQL JOIN with WHERE clause instead of ON. Thanks to Marc + Rechte for the report. + - Fix issue with custom type when multiprocess is used. + - Fix progress bar on final total estimated data in multiprocess mode. + - Fix ORACLE_HOME path in README.md. Thanks to Lubos Cisar for the patch. + - Fix missing replacement with PERFORM in CASE ... WHEN statements. Thanks to + Eric Delanoe for the report. + - Fix duplicate ora2pg command in iteration. + - Improve ora2pg_scanner port on Windows OS. Thanks to Marie Contencin for the + report. + - Add perl call to all ora2pg commands when the scanner is executed on + Windows system as the shebang is not recognized. Thanks to Marie Contencin + for the report. + - Fix several issue with compressed output. Thanks to Bach Nga for the report. + - Fix translation of CURSOR IS SELECT with a comment before the SELECT. + Thanks to Izaak van Niekerk for the report. + - Fix export of procedures as PostgreSQL procedures with version 11. + - Add APEX_040000 to the schemas exclusion list. Thanks to Don Seiler for the + report. + - Fix possible unquoted default values. Thanks to Marc Rechte for the report. + - Fix MySQL SET TRANSACTION clause when TRANSACTION is set to readonly or + readwrite this is not supported so fall back in READ COMMITTED isolation + level in this case. Thanks to Marc Rechte for the report. + - Fix export of functions, column DATA_TYPE does not exists in table + INFORMATION_SCHEMA.ROUTINES before MySQL 5.5.0. Replace it with column + DTD_IDENTIFIER for prior version. Thanks to Marc Rechte for the report. + - Fix double quote in CREATE TRIGGER code and applying of preserve case on + column name. + +2019 01 18 - v20.0 + +This release fix several issues reported during the last three months +and adds several new features and improvement. The change of major +version is related to backward compatibility break with the removed of +most PG_SUPPORTS_* configuration directives and their replacement with +the new PG_VERSION directive. + +New features and configuration directives in this release: + + * Add PG_VERSION configuration directive to set the PostgreSQL major + version number of the target database. Ex: 9.6 or 10. Default is + current major version at time of a new release. This replace the + old PG_SUPPORTS_* configuration directives. + * Removed all PG_SUPPORTS_* configuration directives minus + PG_SUPPORTS_SUBSTR that is related to Redshift engine. + * Export of BFILE as bytea is now done through a PL/SQL function to + extract the content of a BFILE and generate a bytea data suitable + for insert or copy into PostgreSQL. + * Foreign keys that reference a partitioned table are no more + exported. + * Show table name on Oracle side during export using at connection + time: DBMS_APPLICATION_INFO.SET_ACTION(table_name); + * When the date format is ISO and the value is a constant the call + to to_date() is removed and only the constant is preserved. For + example: to_date(' 2013-04-01 00:00:00','SYYYY-MM-DD HH24:MI:SS') + is replaced by a simple call to: ' 2013-04-01 00:00:00'. + This rewrite is limited to PARTITION export type when directive + PG_SUPPORTS_PARTITION is enabled. + * Add DATA_EXPORT_ORDER configuration directive. By default data + export order will be done by sorting on table name. If you have + huge tables at end of alphabetic order and are using multiprocess, + it can be better to set the sort order on size so that multiple + small tables can be processed before the largest tables finish. + In this case set this directive to size. Possible values are name + and size. Note that export type SHOW_TABLE and SHOW_COLUMN will + use this sort order too, not only COPY or INSERT export type. + * Add NO_BLOB_EXPORT configuration directive. Exporting BLOB could + take time and you may want to export all data except the BLOB + columns. In this case enable this directive and the BLOB columns + will not be included into data export. The BLOB column must not + have a NOT NULL constraint. Thanks to Ilya Vladimirovich for the + * Add PREFIX_SUB_PARTITION to enable/disable sub-partitioning table + prefixing in case of the partition names are a part of the sub- + partition names. + * Add special replacement for case of epoch syntax in Oracle: + (sysdate - to_date('01-Jan-1970', 'dd-Mon-yyyy'))*24*60*60 + is replaced by the PostgreSQL equivalent: + (extract(epoch from now())) + +Here is the full list of changes and acknowledgements: + + - Export indexes and constraints on partitioned table with pg >= 11. + - Fix incorrect replacement of NLS_SORT in indexes. + - Bring back DISABLE_UNLOGGED feature. Thanks to Jean-Christophe + Arnu for the patch + - Fix CREATE SCHEMA statement that was not written to dump file. + - Fix DBMS_APPLICATION_INFO.set_action() call, old Oracle version + do not support named parameters. + - Fix duplicate index name on partition. Thanks to buragaddapavan + for the report. + - Add support to new configuration directive PG_VERSION to control + the behavior of Ora2Pg following PostgreSQL version. + - Fix error in creation of default partition with PostgreSQL 10. + Thanks to buragaddapavan for the report. + - Fix missing export of single MAXVALUE partition, this will produce + the following range partition: ... FOR VALUES FROM (MINVALUE) TO + (MAXVALUE) Previous behavior was to not export partition as it is + better to not partition the table at all. However it is declared + in Oracle so it is better to export it to see what can be done. + Thanks to buragaddapavan for the report. + - Do not export foreign keys that reference a partitioned table. + Remove NOT VALID on foreign keys defined on a partitioned + table if present. Thanks to Denis Oleynikov for the report. + - Fix export of BFILE as bytea. Ora2Pg now use a PL/SQL function to + extract the content of a BFILE and generate a bytea data suitable + for insert or copy into PostgreSQL. Thanks to RickyTR for the + report. + - Add TIMEZONE_REGION and TIMEZONE_ABBR to migration assessment, no + direct equivalent in PostgreSQL. Remove NSLSORT not used in + migration assessment. Thanks to buragaddapavan for the report. + - Fix output of multiple export type specified in TYPE directive. + - Rewrite and renaming of _get_sql_data() function into + _get_sql_statements(). + - Limit CURSOR weight in migration assessment to REF CURSOR only, + other case are all covered. REF CURSOR might need a review to see + if they need to be replaced with a SET OF RECORD. + - Fix replacement of EMPTY_CLOB() or EMPTY_BLOB() with empty string + when EMPTY_LOB_NULL is disabled and NULL when it is enabled. + - Prefix output file with the export type in multiple export type + mode, ex: sequence_output.sql or table_output.sql. Thanks to + buragaddapavan for the report. + - Fix export of data from an Oracle nested table. Thanks to rejo + oommen for the report. + - Removed cast to timestamp from partition range. Thanks to + buragaddapavan and rejo-oommen for the report. + - Fix partition default syntax. Thanks to rejo-oommen for the + report. + - Apply missing SYSUSERS schemas exclusion on columns and partition + listing. Thanks to rejo-oommen for the report. + - Add warning about parameter order change in output file. + - Show table name on Oracle side during export using at connection + time: DBMS_APPLICATION_INFO.SET_ACTION(table_name); + Thanks to Denis Oleynikov for the feature request. + - Report change in ORA_RESERVED_WORDS into documentation. + - Add references in the keyword list of ORA_RESERVED_WORDS. + - Fix the missing white space in some lines while creating + import_all.sh file. Thanks to Fabiano for the patch. + - Fix translation of infinity value for float. Thanks to Damien + Trecu for the report. + - Fix default value in timestamp column definition when a timezone + is given. Thanks to buragaddapavan for the report. + - Fix missing export of index and constraint in a partitioned + table when DISABLE_PARTITION is enabled. Thanks to Denis Oleynikov + for the report. + - Prevent PARTITION BY when DISABLE_PARTITION is enabled. Thanks to + Denis Oleynikov for the report. + - Add DATA_EXPORT_ORDER configuration directive. By default data + export order will be done by sorting on table name. If you have + huge tables at end of alphabetic order and are using multiprocess, + it can be better to set the sort order on size so that multiple + small tables can be processed before the largest tables finish. + In this case set this directive to size. Possible values are name + and size. Note that export type SHOW_TABLE and SHOW_COLUMN will + use this sort order too, not only COPY or INSERT export type. + Thanks to Guy Browne for the feature request. + - Fix remove leading ':' on Oracle variable taking care of regex + character class. Thanks to jselbach for the report. + - Add NO_BLOB_EXPORT configuration directive. Exporting BLOB could + take time and you may want to export all data except the BLOB + columns. In this case enable this directive and the BLOB columns + will not be included into data export. The BLOB column must not + have a NOT NULL constraint. Thanks to Ilya Vladimirovich for the + feature request. + - Fix incorrect rewrote of the first custom type in a row. Thanks + to Francesco Loreti for the patch. + - Remove double quote in type definition en set type name in lower + case when PRESERVE_CASE is disabled. + - Add PREFIX_SUB_PARTITION to enable/disable sub-partitioning table + prefixing in case of the partition names are a part of the sub- + partition names. + - Fix epoch replacement case in CREATE TABLE statements. + - Apply epoch replacement to default value in table declaration. + - Add special replacement for case of epoch syntax in Oracle: + (sysdate - to_date('01-Jan-1970', 'dd-Mon-yyyy'))*24*60*60 + is replaced by the PostgreSQL equivalent: + (extract(epoch from now())) + Thanks to rejo-oommen for the feature request. + - A few typos in --help sections. Thanks to Christophe Courtois + for the report. + - Fix export of primary key on partition table. Thanks to chmanu + for the patch. + - Fix malformed user defined type export. Thanks to Francesco Loreti + for the report. + + +2018 09 27 - v19.1 + +This release fix several issues reported during the last month and +add support to PostgreSQL 11 HASH partitioning. + +It also adds some new features and configuration directives: + + * Add export of default partition and default sub partition. + * Add export of HASH partition type. + * Add support of stored procedure object. + * Add replacement of NLSORT in indexes or queries. For example: + CREATE INDEX test_idx ON emp + (NLSSORT(emp_name, 'NLS_SORT=GERMAN')); + is translated into + CREATE INDEX test_idx ON emp + ((emp_name collate "german")); + The collation still need to be adapted, here probably "de_DE". + NLSSORT() in ORDER BY clause are also translated. + * Prevent duplicate index with primary key on partition to be + exported. + * PostgreSQL native partitioning does not allow direct import of + data into already attached partitions. We now force direct import + into main table but we keep Oracle export of data from individual + +This release also adds two new command line options: + + --oracle_speed: use to know at which speed Oracle is able to send + data. No data will be processed or written + --ora2pg_speed: use to know at which speed Ora2Pg is able to send + transformed data. Nothing will be written + +Use it for debugging purpose. They are useful to see Oracle speed to +send data and at what speed Ora2Pg is processing the data without +reaching disk or direct import into PostgreSQL. + +Two new configuration directive has been added: + + * PG_SUPPORTS_PROCEDURE : PostgreSQL v11 adds support to stored + procedure objects. Disabled by default. + - PARALLEL_MIN_ROWS: set the minimum number of tuples in a table + before calling Oracle's parallel mode during data export. + Default to 100000 rows. + +Note that PG_SUPPORTS_PARTITION and PG_SUPPORTS_IDENTITY are now +enabled by default to use PostgreSQL declarative partionning and +identity column instead of serial data type. + +Here is the full list of changes and acknowledgements: + + - Fix automatic quoting of table or partition name starting with + a number. Thanks to Barzaqh for the report. + - Add information about custom directory installation. Thanks to + joguess for the report. + - Update list of action in documentation. + - Fix export of spatial geometries. Thanks to burak yurdakul for + the report. + - Fix translation of default value in CREATE TABLE DDL when using + a function. Thanks to Denis Oleynikov for the report. + - Prevent moving index on partition during tablespace export. + Thanks to Maxim Zakharov for the report. + - Fix upper case of partition name in triggers. + - Enforce KEEP_PKEY_NAMES when USE_TABLESPACE is enabled. Thanks + to Maxim Zakharov for the patch. + - Fix parsing of Oracle user login in dblink input from a file. + - Fix multiple duplication of range clause in partition export. + - Add bench of total time and rows to migrate data from Oracle + in debug mode with speed average. + - Fix sub partition prefix name. + - Fix unset oracle username when exporting DBLINK from database. + Thanks to Denis Oleynikov for the report. + - Remove NO VALID to foreign keys on partitioned table. Thanks to + Denis Oleynikov for the report. + - Fix crash of Ora2Pg on regexp with dynamic pattern base on package + code. Thank to Alain Debie and MikeCaliffCBORD for the report. + - PostgreSQL native partitioning does not allow direct import of + data into already attached partitions. When PG_SUPPORTS_PARTITION + is enable we now force direct import into main single table but + we keep Oracle export of data from individual partition. Previous + behavior was to use main table from both side. Thanks to Denis + Oleynikov for the report. + - Add the PARALLEL_MIN_ROWS configuration directive to prevent + Oracle's parallel mode to be activated during data export if the + table have less than a certain amount of rows. Default is 100000 + rows. This prevent unnecessary fork of Oracle process. Thanks to + Denis Oleynikov for the feature request. + - Fix composite partition MODULUS value. Thanks to Denis Oleynikov + for the report. + - Fix count of partitions that was not including subpartition count. + - Force PostgreSQL user in FDW user mapping to be PG_USER when it is + defined. + - Sometimes Oracle indexes can be defined as follow: + CREATE INDEX idx_err_status_id + ON err_status (status_id, 1); + which generate errors on PostgreSQL. Remove column names composed + of digit only from the translation. Thanks to Denis Oleynikov for + the report. + - Move Oracle indexes or PK defined on partitioned tables to each + partition as PostgreSQL do not support UNIQUE, PRIMARY KEY, + EXCLUDE, or FOREIGN KEY constraints on partitioned tables. + Definition are created in file PARTITION_INDEXES_output.sql + generated with the PARTITION export type. Thanks to Denis + Oleynikov for the feature request. + - Fix parallel data load from Oracle partitioned tables by using + a unique alias. Thanks to Denis Oleynikov for the report. + - Fix export of composite partitioned (range/hash) table when + PG_SUPPORTS_PARTITION is disabled. Thanks to Denis Oleynikov + for the report. + - Remove composite sub partition from the list of partition, this + return a wrong partition count. + - Fix MODULUS value in hash sub partitioning. + - Index and table partitions could be on separate tablespaces. + Thanks to Maxim Zakharov for the patch. + - Fix case where procedure object name is wrongly double quoted. + Thanks to danghb for the report. + - Fix parser to support comment between procedure|function name + and IS|AS keyword. Thanks to danghb for the report. + - Remove dependency to List::Util for the min() function. + + +2018 08 18 - v19.0 + +This major release fix several issues reported by users during last +year. It also adds several new features and configuration directives. + +New features: + + - Add export of Oracle HASH partitioning when PG_SUPPORTS_PARTITION + is enabled. This is a PostgreSQL 11 feature. + - Add SUBTYPE translation into DOMAIN with TYPE and PACKAGE export. + - Add automatic translation of + KEEP (DENSE_RANK FIRST|LAST ORDER BY ...) OVER (PARTITION BY ...) + into + FIRST|LAST_VALUE(...) OVER (PARTITION BY ... ORDER BY ...). + - Add PCTFREE to FILLFACTOR conversion when PCTFREE is upper than + the default value: 10. + - Replace DELETE clause not followed with FROM (optional in Oracle). + - Remove Oracle extra clauses in TRUNCATE command. + - Allow use of NUMBER(*) in DATA_TYPE directive to convert all + NUMBER(*) into the given type whatever is the length. Ex: + DATA_TYPE NUMBER(*):bigint. + - Add a PARALLEL hint to all Oracle queries used to migrate data. + - Add export of Identity Columns from Oracle Database 12c. + - Add translation of UROWID datatype and information in documentation + about why default corresponding type OID will fail at data import. + - Remove unwanted and unused keywords from CREATE TABLE statements: + PARALLEL and COMPRESS. + - Remove TEMPORARY in DROP statements. + - Improve speed of escape_copy() function used for data export. + - Add translation of Oracle functions NUMTOYMINTERVAL() and + NUMTODSINTERVAL(). + - Add counting of jobs defined in Oracle scheduler in the migration + assessment feature. + - Add CSMIG in the list of Oracle default system schema + - Fully rewrite data export for table with nested user defined types + DBD::Oracle fetchall_arrayref() is not able to associate complex + custom types to the returned arrays, changed this call to use + fetchrow_array() also used to export BLOB. + - QUERY export will now output translated queries as well as + untranslated ones. This break backward compatibility, previously + only translated query was dumped. + - Auto detect UTF-8 input files to automatically use utf8 encoding. + - Support translation of MySQL global variables. + - Add translation of preprocessor in Oracle external table into + program in foreign table definition. Allow translation of external + table from file. + - Add translation to NVL2() Oracle function. + - Translate CONVERT() MySQL function. + - Translate some form of GROUP_CONCAT() that was not translated. + - Remove call to CHARSET in cast() function, replace it by COLLATE + every where else. This must cover most of the cases but some + specials use might not, so please reports any issue with this + behavior. + - Add -c | --config command line option to ora2pg_scanner to set + custom configuration file to be used instead of ora2pg default: + /etc/ora2pg/ora2pg.conf + - Improve CONNECT BY and OUTER JOIN translation. + - And lot of MySQL to PostgreSQL improvements. + +Several new configuration directives have been added: + + - Add DEFAULT_PARALLELISM_DEGREE to control PARALLEL hint use + when exporting data from Oracle. Default is disabled. + - Make documentation about KEEP_PKEY_NAMES more explicit about + kind of constraints affected by this directive. + - Add PG_SUPPORTS_IDENTITY configuration directive to enable + export of Oracle identity columns into PostgreSQL 10 feature. + If PG_SUPPORTS_IDENTITY is disabled and there is IDENTITY column + in the Oracle table, they are exported as serial or bigserial + columns. When it is enabled they are exported as IDENTITY columns + like: + + CREATE TABLE identity_test_tab ( + id bigint GENERATED ALWAYS AS IDENTITY, + description varchar(30) + ) ; + + If there is non default sequence option set in Oracle, they will + be appended after the IDENTITY keyword. Additionally in both cases + Ora2Pg will create a file AUTOINCREMENT_output.sql with a function + to update the associated sequences with the restart value set to + "SELECT max(colname)+1 FROM tablename". Of course this file must + be imported after data import otherwise sequence will be kept to + start value. + - Add DISABLE_UNLOGGED configuration directive. By default Ora2Pg + export Oracle tables with the NOLOGGING attribute into UNLOGGED + tables. You may want to fully disable this feature because you + will lost all data from unlogged table in case of PostgreSQL crash. + Set it to 1 to export all tables as normal table. When creating a + new migration project using --init_project, this directive is + activated by default. This is not the case in the default + configuration file for backward compatibility. + - Add FORCE_SECURITY_INVOKER configuration directive. Ora2Pg use + the function's security privileges set in Oracle and it is often + defined as SECURITY DEFINER. To override those security privileges + for all functions and use SECURITY DEFINER instead, enable this + directive. + - Add AUTONOMOUS_TRANSACTION in configuration to enable translation + of autonomous transactions into a wrapper function using dblink + or pg_background extension. If you don't want to use this feature + and just want to export the function as a normal one without the + pragma call, disable this directive. + - Add documentation about COMMENT_SAVEPOINT configuration directive. + - Major rewrite in PACKAGE parser to better support global variables + detection. Global variable that have no default values are now + always initialized to empty string in file global_variables.conf + so that we see that they exists. This might not change the global + behavior. + +I especially want to thank Pavel Stehule and Eric Delanoe who spent +lot of time this year to help me to improve the PL/SQL to plpgsql +translation and also Krasiyan Andreev who help a lot to finalize +the MySQL to PostgreSQL migration features. + +Here is a complete list of changes and acknowledgments: + + - Fix translation of "varname cursor%ROWTYPE;". Thanks to Philippe + Beaudoin for the report. + - Fix return of autonomous transaction dblink call when function has + OUT parameter. Thanks to Pavel Stehule for the report. + - Add Oracle to PostgreSQL translation of windows functions + KEEP (DENSE_RANK FIRST|LAST ORDER BY ...) OVER (PARTITION BY ...) + Thanks to Swapnil bhoot929 for the feature request. + - Fix "ORA-03113: end-of-file on communication channel" that what + generated by a too long query send to Oracle. The size of queries + sent to Oracle to retrieve object information depend of the ALLOW + and EXCLUDE directives. If you have lot of objects to filter you + can experience this kind of non explicit error. Now Ora2pg use + bind parameter to pass the filters values to reduce the size of + the prepared query. Thanks to Stephane Tachoire for the report. + - Add SUBTYPE translation into DOMAIN with TYPE and PACKAGE export. + Thanks to Francesco Loreti for the feature request. + - Fix PLS_INTEGER replacement. + - Remove precision for RAW|BLOB as type modifier is not allowed for + type "bytea". + - Fix call of schema.pckg.function() in indexes with a replacement + with pckg.function(). Thanks to w0pr for the report. + - Fix translation of UPDATE trigger based on columns: + "BEFORE UPDATE OF col1,col2 ON table". + Thanks to Eric Delanoe for the report. + - Remove single / from input file that was causing a double END in + some case. Thanks to Philippe Beaudoin for the report. + - Limit translation of PCTFREE into FILLFACTOR when PCTFREE is upper + than the Oracle default value: 10. With PostgreSQL 100 (complete + packing) is the default. + - Add PCTFREE to FILLFACTOR conversion. Thanks to Maxim Zakharov + for the patch. + - Remove TRUNCATE extra clauses. Thanks to e7e6 for the patch. + - Fix type conversion when extra \n added after ;. Thanks to + Maxim Zakharov for the patch. + - Fix DELETE clause not followed with FROM (optional in Oracle). + Thanks to Philippe Beaudoin for the patch. + - Limit call to ALL_TAB_IDENTITY_COLS to version 12+. Thanks to + Andy Garfield for the report. + - Fix comment parsing. Thanks to Philippe Beaudoin for the report. + - Allow use of NUMBER(*) in DATA_TYPE directive to convert all + NUMBER(*) into the given type whatever is the length. + Thanks to lingeshpes for the feature request. + - Fix bug in function-based index export. Thanks to apol1234 for + the report. + - Add PARALLEL hint to all data export queries. Thanks to jacks33 + for the report. + - Make documentation about KEEP_PKEY_NAMES more explicit about kind + of constraints affected by this directive. + - Fix export of identity columns by enclosing options between + parenthesis and replacing CACHE 0 by CACHE 1. Thanks to swmcguffin + devtech for the report. + - Add parsing of identity columns from file. + - Fix unwanted replacement of IF () in MySQL code. Thanks to + Krasiyan Andreev for the report. + - Fix to_char() translation, thanks to Eric Delanoe for the report. + - Fix untranslated PERFORM into exception. Thanks to Pavel Stehule + for the report. + - Add _get_entities() function to MySQL export. It returns nothing, + AUTO_INCREMENT column are translated with corresponding types, + smallserial/serial/bigserial. + - Fix look at encrypted column on Oracle prior to 10. Thanks to + Stephane Tachoires for the patch. + - Add export of Identity Columns from Oracle Database 12c. Thanks + to swmcguffin-devtech for the feature request. + - Prevent Ora2Pg to scan ALL_SCHEDULER_JOBS for version prior to 10 + Thanks to Stephane Tachoires for the patch. + - Fix pull request #648 to log date only when debug is enabled and + use POSIX strftime instead of custom gettime function. + - Add system time to debug log info. Thanks to danghb for the patch. + - Fix parsing of trigger from file and exception. + - Fix very slow export of mysql tablespace when number of table is + large. Thanks to yafeishi for the report. + - Fix translation of CAST( AS unsigned). Thanks to Krasiyan Andreev. + - Fix MySQL character length to use character_maximum_length + instead of equal character_octet_length. Thanks to yafeishi for + the report. + - Fix custom replacement of MySQL data type. Thanks to Krasiyan + Andreev for the report. + - Fix replacement of call to open cursor with empty parenthesis. + Thanks to Philippe Beaudoin for the report. + - Fix MySQL data type conversion in function declaration. Thanks to + Krasiyan Andreev for the report. + - Fix error with -INFINITY as default value for date or timestamp + columns. + - Fix procedure call rewrite with unwanted comma on begin of + parameter list. Thanks to Pavel Stehule for the report. + - Fix handling of foreign keys when exporting data and DROP_FKEYS + is enabled and ALLOW/EXCLUDE directive is set. Now Ora2Pg will + first drop all foreign keys of a table in the export list and all + foreign keys of other tables pointing to the table. After data + import, it will recreate all of these foreign keys. Thanks to + Eric Delanoe for the report. + - Fix broken transformation of procedure call with default parameter + Thanks to Pavel Stehule for the report. + - Translate call to TIMESTAMP in partition range values into a cast. + Thanks to markiech for the report. + - Fix CONNECT BY translation when the query contain an UNION. Thanks + to mohammed-a-wadod for the report. + - Fix CONNECT BY with PRIOR on the right side of the predicat. + - Fix outer join translation when the (+) was in a function, ex: + WHERE UPPER(trim(VW.FRIDAY))= UPPER(trim(FRIDAY.NAME(+))). + - Order outer join pending tables in from clause. + - Order by object name comments and indexes export. + - Fix outer join translation when the table is not in the from + clause. Thanks to Cyrille Lintz for the report. + - Try to fix potential Oracle schema prefixing PostgreSQL schema + name in CREATE SCHEMA. Thanks to Cyrille Lintz for the report. + - Fix error in TRIM() translation. Thanks to Cyrille Lintz for the + report. + - Add translation of UROWID datatype and information in documentation + about why default corresponding type OID will fail at data import. + Thanks to Cyrille Lintz for the report. + - Fix bug in exporting boolean default values in column definition. + - Fix bug in column parsing in CREATE TABLE. + - Adapt default value for data type changed to boolean. + - Fix bad handling of -D (data_type) option. + - Change behavior in the attempt to set MySQL global variable type. + Now variable type will be timestamp if the variable name contains + datetime, time if the name contains only time and date for date. + Thanks to Krasiyan Andreev for the report. + - Fix function replacement in MySQL declare section. Thanks to + Krasiyan Andreev fr the report. + - Apply REPLACE_ZERO_DATE to default value in table declaration. + Thanks to Krasiyan Andreev for the report. + - Add support to embedded comment in table DDL. + - Fix replacement of data type for MySQL code. Thanks to Krasiyan + Andreev for the report. + - Fix MySQL type replacement in function. Thanks to Krasiyan Andreev + for the report. + - Improve speed of escape_copy() function used for data export. + Thanks to pgnickb for the profiling. + - Add translation of Oracle functions NUMTOYMINTERVAL() and + NUMTODSINTERVAL(). Thanks to Pavel Stehule for the report. + - Counting jobs defined in Oracle scheduler. Thanks to slfbovey + for the patch. + - Fix several issue in create table DDL parser: + - remove double quote of object name when a list of column is + entered + - split of table definition to extract column and constraint + parts is now more efficient + - remove dot in auto generated constraint name when a schema + is given in table name + - fix default values with space that was breaking the parser + - Remove use of bignum perl module that reports error on some + installation. Thanks to Cyrille Lintz for the report. + - Fix a typo preventing perldoc to complete. Thanks to slfbovey + for the patch. + - Fully rewrite data export for table with nested user defined types + DBD::Oracle fetchall_arrayref() is not able to associate complex + custom types to the returned arrays, changed this call to use + fetchrow_array() also used to export BLOB. Thanks to lupynos for + the report. + - Fix renaming of temporary files during partitions data export. + - Fix Oracle use of empty string as default value for integers. + Oracle allow such declaration: SOP NUMBER(5) DEFAULT '' which + PostgreSQL does not support. Ora2Pg now detect this syntax and + replace empty string with NULL. Thanks to ricdba for the report. + - Add detection of Oracle version before setting datetime format, + needed for Oracle 8i compatibility. + - Export of tables from Oracle database are now ordered by name by + default. Thanks to Markus Roth for the report. + - Fix an other case of missing translation of UNSIGNED into bigint. + Thanks to Krasiyan Andreev for the report. + - Force replacement of double quote into single quote for MySQL view + and function code. + - Fix case when SET @varname := ... is used multiple time in the + same function. Thanks to Krasiyan Andreev for the report. + - Fix case where SET @varname := ... was not translated. Thanks to + Krasiyan Andreev for the report. + - Adjust the regex pattern of last patch. + - Fix unwanted newline after hint replacement that could break + comments. Thanks to Pavel Stehule for the report. + - Fix if() replacement in query. Thanks to Krasiyan Andreev for the + report. + - Remove extra parenthesis in some form of JOIN. Thanks to Krasiyan + Andreev for the report. + - Fix untranslated call to UNSIGNED, now translated as bigint. + - Thanks to Krasiyan Andreev for the report. + - Fix translation of double(p,s) into decimal(p,s). + - Remove use of SET when an assignment is done through a SELECT + statement. Thanks to Krasiyan Andreev for the report. + - Fix non-quoted reserved keywords in INSERT / COPY statements when + exporting data. Thanks to Pavel Stehule for the report. + - Fix partition data export to file, temporary files for partition + output was not renamed at export end then data was not loaded. + - Fix double operator := during function with out param rewrite. + - Fix commit f1166e5 to apply changes when FILE_PER_FUNCTION is + disable or when an input file is given. + - Fix translation of LOCATE(). Thanks to Krasiyan Andreev for the + report. + - Fix case where MySQL GROUP_CONCAT() function was not translated. + Thanks to Krasiyan Andreev for the report. + - Fix :new and :old translation in triggers. + - Fully rewrite function call qualification process, the second pass + now is only use to requalify call to pkg.fct into pkg_ftc when + PACKAGE_AS_SCHEMA is disable. The replacement of all function + calls using double quote when a non supported character is used or + when PRESERVE_CASE is enabled has been completely removed as this + takes too much time to process for just very few case. So by + default now Ora2Pg will not go through the second pass. This can + change in the future especially if this is more performant to + process PERFORM replacement. Thanks a lot to Eric Delanoe for his + help on this part. + - Exclude function and procedure not from package to be used in + requalify call. Thanks to Eric Delanoe for the report. + - Fix function name qualification in multiprocess mode. + - Fix unqualified function call due to unclose file handle. + - Prevent try to requalify function call if the function is + not found in the file content. + - Remove ALGORITHM=.*, DEFINER=.* and SQL SECURITY DEFINER from + MySQL DDL code. + - An other missing change to previous commit on qualifying function + call. + - Limit function requalification to export type: VIEW, TRIGGER, + QUERY, FUNCTION, PROCEDURE and PACKAGE. + - Auto detect UTF-8 input files to automatically use utf8 encoding. + - Remove all SHOW ERRORS and other call to SHOW in Oracle package + source as they was badly interpreted as global variable. + - Fix MySQL CREATE TABLE ... SELECT statement. + - Fix pending translation issue on some DATE_FORMAT() case. + Thanks to Krasiyan Andreev for the report. + - Fix translation of IN (..) in MySQL view. Thanks to Krasiyan + Andreev for the report. + - Fix MySQL date format with digit. + - Fix DATE_FORMAT, WHILE and IFNULL translation issues. + - Fix not translated MySQL IF() function. + - Fix other MySQL translation issues for @variable. Thanks to + Krasiyan Andreev for the report. + - Fix issue in MySQL IF translation with IN clause. Thanks to + Krasiyan Andreev for the report. + - Clarify comment about XML_PRETTY directive. Thanks to TWAC + for the report. + - Fix remaining MySQL translation issues for @variable reported + in issue #590. + - Fix no translated := in SET statement. + - Fix output order of translated function. + - Fix non printable character or special characters that make + file encoding to ISO-8859 instead of utf8. Thanks to twac for + the report. + - Prevent MySQL global variable to be declared twice. Thanks to + Krasiyan Andreev for the report. + - Support translation of MySQL global variables. Session variable + @@varname are translated to PostgreSQL GUC variable and global + variable @varname are translated to local variable defined in a + DECLARE section. Ora2Pg tries to gather the data type by using + integer by default, varchar if there is a constant string ('...') + in the value and a timestamp if the variable name have the keyword + date or time inside. Thanks to Krasiyan Andreev for the feature + request. + - Fix DATE_ADD() translation. + - Add translation of preprocessor in Oracle external table into + program in foreign table definition. Thanks to Thomas Reiss for + the report. Allow translation of external table from file. + - Fix case where IF EXISTS might not be append when it is not + supported by PG. + - Translate CONVERT() MySQL function. Thanks to Krasiyan Andreev + for the report. + - Translate some form of GROUP_CONCAT() that was not translated. + Thanks to Krasiyan Andreev for the report. + - Apply same principe with COMMIT in MySQL function code than in + Oracle code. It is kept untouched to be able to detect a possible + change of code logic. It can be automatically commented if + COMMENT_COMMIT_ROLLBACK is enabled. Also I have kept the START + TRANSACTION call but it is automatically commented. + - Add mysql_enable_utf8 => 1 to MySQL connection to avoid issues + with encoding. Thanks to Krasiyan Andreev for the report. + - Prevent removing of comment on MySQL function and add a "COMMENT + ON FUNCTION" statement at end of the function declaration. Thanks + to Krasiyan Andreev for the report. + - Fix translation of types in MySQL function parameter. Thanks to + Krasiyan Andreev for the report. + - Remove START TRANSACTION from MySQL function code. Thanks to + Krasiyan Andreev for the report. + - Fix previous patch, we do not need to look forward for function + or procedure definition in VIEW export and there is no package + with MySQL. Thanks to Krasiyan Andreev for the report. + - Fix call to useless function for MySQL function. + - Add rewrite of MySQL function call in function or procedure code + translation and some other translation related to MySQL code. + - Fix ora2pg_scanner when exporting schema with $ in its name. + Thanks to Aurelien Robin for the report. + - Disable number of microsecond digit for Oracle version 9. Thanks + to Aurelien Robin for the report. + - Do not look at encrypted column for DB version < 10. Thanks to + Aurelien Robin for the report. + - Fix MySQL call to charset in cast function. MySQL charset "utf8" + is also set to COLLATE "C.UTF-8". Thanks to Krasiyan Andreev for + the report. + - Fix two bug in CONNECT BY and OUTER JOIN translation. + - Forgot to handle exception to standard call to IF in MySQL IF() + translation. Thanks to Krasiyan Andreev for the report. + - Forgot to apply previous changes to procedure. + - Fix IF() MySQL replacement when nested and when containing an + IN (...) clause. Thanks to Krasiyan Andreev for the report. + - Fix double BEGIN on MySQL function export. Thanks to Krasiyan + Andreev for the report. + - Fix enum check constraint name when PRESERVE_CASE is enabled. + - Fix case where object with LINESTRING and CIRCULARSTRING was + exported as MULTILINESTRING instead of MULTICURVE. + - Fix export of MULTICURVE with COMPOUNDCURVE. Thanks to Petr Silhak + for the report. + - Fix several issue in MySQL table DDL export. Thanks to Krasiyan + Andreev for the report. + - Fix MySQL auto_increment data type translation and columns export + order. + - Fix translation of MySQL function CURRENT_TIMESTAMP(). Thanks to + Krasiyan Andreev for the report. + - Fix export of MySQL alter sequence name when exporting auto + increment column. Thanks to Krasiyan Andreev for the report. + - Replace IF() call with CASE ... END in VIEW and QUERY export for + MySQL. Thanks to Krasiyan Andreev for the feature request. + - Replace backquote with double quote on mysql statements when read + from file. + - Fix bug in REGEXP_SUBSTR replacement. + - Prevent replacement with same function name from an other package. + Thanks to Eric Delanoe for the report. + - Apply same STRICT rule for SELECT INTO to EXECUTE INTO. Thanks to + Pavel Stehule for the report. + - Fix extra parenthesis removing when a OR clause is present. Thanks + to Pavel Stehule for the report. + - Keep autonomous pragma commented when conversion is deactivated + to be able to identify functions using this pragma. + - Fix bug in replacement of package function in string constant. + - Fix malformed replacement of array element calls. Thanks to Eric + Delanoe for the report. + - Fix unwanted replacement of TO_NUMBER function. Thanks to Torquem + for the report. + - Add an example of DSN for MySQL in ORACLE_DSN documentation. + Thanks to François Honore for the report. + - Fix typo in default dblink connection string. Thanks to Pavel + Stehule for the report. + - Add information about Oracle Instant Client installation. Thanks + to Jan Birk for the report. + - Replace Oracle array syntax arr(i).x into arr[i].x into PL/SQL + code. Thanks to Eric Delanoe for the report. + - Use a more generic connection string for DBLINK. It will use + unix socket by default to connect and the password must be set + in .pgpass. This will result in the following connection string: + format('port=%s dbname=%s user=%', current_setting('port'), + current_database(), current_user) + If you want to redefine this connection string use DBLINK_CONN + configuration directive. Thanks to Pavel Stehule for the feature + request. + - Fix missing RETURN NEW in some trigger translation. Thanks to + Pavel Stehule for the report. + - Fix a missing but non mandatory semi-comma. + - Keep PKs/unique constraints which are deferrable in Oracle also + deferrable in PostgreSQL. Thank to Sverre Boschman for the patch. + - Fix parsing and translation of CONNECT BY. Thanks to bhoot929 + for the report. + - Fix FDW export when exporting all schema. Thanks to Laurenz Albe + for the report. + - Add a note about multiple value in export type that can not + include COPY or INSERT together with others export type. + - Fix duplicate condition. Thanks to Eric Delanoe for the report. + - Fix unwanted translation into PERFORM after INTERSECT. + - Comment savepoint in code. Thanks to Pavel Stehule for the patch. + - Fix "ROLLBACK TO" that was not commented. Thanks to Pavel Stehule + for the report. + - Fix restore of constant string when additional string constant + regex are defined in configuration file. + - Fix translation of nextval with sequence name prefixed with their + schema. + - Cast call to TO_DATE(LOCALTIMESTAMP,...) translated into + TO_DATE(LOCALTIMESTAMP::text,...). Thanks to Keshav kumbham + for the report. + - Remove double quote added automatically by Oracle on view + definition when PRESERVE_CASE is not enable. Thanks to JeeIPI for + the report. + - Fix translation of FROM_TZ with a call to function as first + parameter. Thanks to TrungPhan for the report. + - Fix package export when FILE_PER_FUNCTION is set. Thanks to + Julien Rouhaud for the report. + - Add translation of REGEXP_SUBSTR() with the following rules: + Translation of REGEX_SUBSTR( string, pattern, [pos], [nth]) + converted into + SELECT array_to_string(a, '') + FROM regexp_matches(substr(string, pos), pattern, 'g') + AS foo(a) + LIMIT 1 OFFSET (nth - 1); + Optional fifth parameter of match_parameter is appended to 'g' + when present. Thanks to bhoot929 for the feature request. + - Add count of REGEX_SUBSTR to migration assessment cost. + - Add translation support of FROM_TZ() Oracle function. Thanks + to trPhan for the feature request. + - Forces ora2pg to output a message when a custom exception code + has less than 5 digit. + - Fix errcode when Oracle custom exception number have less than + five digit. Thanks to Pavel Stehule for the report. + - Fix case where custom errcode are not converted. Thanks to Pavel + Stehule for the report. + - Fix print of single semicolon with empty line in index export. + - Fix problem with TO_TIMESTAMP_TZ conversion. Thanks to Keshav- + kumbham for the report. + - Fix unwanted double quote in index column with DESC sorting. + Thanks to JeeIPI for the report. + - Fix non detection case of tables in from clause for outer join + translation. Thanks to Keshav for the report. + - Fix unwanted replacement of = NULL into IS NULL in update + statement. Thanks to Pavel Stehule for the report. + - Force schema name used in TEST action to lowercase. Thanks to + venkatabn for the report. + - Fix export of spatial geometries with CURVEPOLYGON + COMPOUNDCURVE + Thanks to kabog for the report. + +2017 09 01 - v18.2 + +This release fix several issues reported during the last six months. +It also adds several new features and configuration directives: + + - Add translation of SUBSTRB into substr. + - Allow use of array in MODIFY_TYPE to export Oracle user defined + type that are just array of some data type. For example: + CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); + can be directly translated into text[] or varchar[]. In this case + use the directive as follow: MODIFY_TYPE CLUB:MEMBERS:text[] + Ora2Pg will take care to transform all data of this column into + the correct format. Only arrays of characters and numerics types + are supported. + - Add translation of Oracle function LISTAGG() into string_agg(). + - Add TEST_VIEW action to perform a simple count of rows returned by + views on both database. + - Translate SQL%ROWCOUNT into GET DIAGNOSTICS rowcount = ROW_COUNT + and add translation of SQL%FOUND. + - Add translation of column in trigger event test with IS DISTINCT, + for example: IF updating('ID') THEN ... will be translated into: + IF TG_OP = 'UPDATE' AND NEW.'ID' IS DISTINCT FROM OLD.'ID' then... + - Replace UTL_MATH.EDIT_DISTANCE function by fuzzymatch levenshtein. + - Allow use of MODIFY_STRUCT with TABLE export. Table creation DDL + will respect the new list of columns and all indexes or foreign + key pointing to or from a column removed will not be exported. + - Add export of partition and subpartition using PostgreSQL native + partitioning. + - Auto detect encrypted columns and report them into the assessment. + SHOW_COLUMN will also mark columns as encrypted. + - Add information to global temporary tables in migration assessment. + - Add experimental DATADIFF functionality. + - Allow use of multiprocess with -j option or JOBS to FUNCTION and + PROCEDURE export. Useful if you have thousands of these objects. + - Force RAW(N) type with default value set to sys_guid() as UUID + on PostgreSQL. + - Replace function with out parameter using select into. For example + a call to: get_item_attr( attr_name, p_value ); + where p_value is an INOUT parameter, will be rewritten as + + p_value := get_item_attr( attr_name, p_value ); + + If there is multiple OUT parameters, Ora2Pg will use syntax: + + SELECT get_item_attr( attr_name, p_value ) + INTO (attr_name, p_value); + + - Add translation of CONNECT BY using PostgreSQL CTE equivalent. + This translation also include a replacement of LEVEL and + SYS_CONNECT_BY_PATH native Oracle features. On complex queries + there could still be manual editing but all the main work is done. + - Add support to user defined exception, errcode affected to each + custom exception start from 50001. + - Translate call to to_char() with a single parameter into a cast + to varchar. Can be disabled using USE_ORAFCE directive. + - Improve ora2pg_scanner to automatically generates migration + assessment reports for all schema on an Oracle instance. Before + the schema name to audit was mandatory, now, when the schema + is not set Ora2Pg will scan all schema. The connexion user need + to have DBA privilege. Ora2Pg will also add the hostname and SID + as prefix in the filename of the report. This last change forbids + ora2pg_scanner to overwrite a report if the same schema name is + found in several databases. + +Several new configuration directives have been added: + + - Add USE_ORAFCE configuration directive that can be enabled if you + want to use functions defined in the Orafce library and prevent + Ora2Pg to translate call to these functions. The Orafce library + can be found here: https://github.com/orafce/orafce + By default Ora2pg rewrite add_month(), add_year(), date_trunc() + and to_char() functions, but you may prefer to use the Orafce + functions that do not need any code transformation. Directive + DATE_FUNCTION_REWRITE has been removed as it was also used to + disable replacement of add_month(), add_year() and date_trunc() + when Orafce is used, useless now. + - Add FILE_PER_FKEYS configuration directive to allow foreign key + declaration to be saved in a separate file during schema export. + By default foreign keys are exported into the main output file or + in the CONSTRAINT_output.sql file. If enabled foreign keys will be + exported into a file named FKEYS_output.sql + - Add new COMMENT_COMMIT_ROLLBACK configuration directive. Call to + COMMIT/ROLLBACK in PL/SQL code are kept untouched by Ora2Pg to + force the user to review the logic of the function. Once it is + fixed in Oracle source code or you want to comment this calls + enable the directive. + - Add CREATE_OR_REPLACE configuration directive. By default Ora2Pg + use CREATE OR REPLACE in function DDL, if you need not to override + existing functions disable this configuration directive, DDL will + not include OR REPLACE. + - Add FUNCTION_CHECK configuration directive. Disable this directive + if you want to disable check_function_bodies. + + SET check_function_bodies = false; + + It disables validation of the function body string during CREATE + FUNCTION. Default is to use de postgresql.conf setting that enable + it by default. + - Add PG_SUPPORTS_PARTITION directive, disabled by default. + PostgreSQL version prior to 10.0 do not have native partitioning. + Enable this directive if you want to use PostgreSQL declarative + partitioning instead of the old style check constraint and trigger. + - Add PG_SUPPORTS_SUBSTR configuration directive to replace substr() + call with substring() on old PostgreSQL versions or some fork + like Redshift. + - Add PG_INITIAL_COMMAND to send some statements at session startup. + This directive is the equivalent used for Oracle connection, + ORA_INITIAL_COMMAND. Both can now be used multiple time now. + - Add DBLINK_CONN configuration directive. By default if you have + an autonomous transaction translated using dblink extension the + connection is defined using the values set with PG_DSN, PG_USER + and PG_PWD. If you want to fully override the connection string + use this directive to set the connection in the autonomous + transaction wrapper function. For example: + + DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass + + - Add STRING_CONSTANT_REGEXP configuration directive. Ora2Pg replace + all string constant during the pl/sql to plpgsql translation, + string constant are all text include between single quote. If you + have some string placeholder used in dynamic call to queries you + can set a list of regexp to be temporary replaced to not break the + parser. For example: + + STRING_CONSTANT_REGEXP + + The list of regexp must use the semi colon as separator. + - Add FUNCTION_STABLE configuration directive. By default Oracle + functions are marked as STABLE as they can not modify data unless + when used in PL/SQL with variable assignment or as conditional + expression. You can force Ora2Pg to create these function as + VOLATILE by disabling this configuration directive. + - Add new TO_NUMBER_CONVERSION configuration directive to control + TO_NUMBER translation behavior. By default Oracle call to function + TO_NUMBER will be translated as a cast into numeric. For example, + TO_NUMBER('10.1234') is converted into PostgreSQL call: + to_number('10.1234')::numeric. + If you want you can cast the call to integer or bigint by changing + the value of the configuration directive. If you need better + control of the format, just set it as value, for example: + TO_NUMBER_CONVERSION 99999999999999999999D9999999999 + will convert the code above as: + TO_NUMBER('10.1234', '99999999999999999999D9999999999') + Any value of the directive that it is not numeric, integer or + bigint will be taken as a mask format. If set to none, then no + conversion will be done. + - Add LOOK_FORWARD_FUNCTION configuration directive which takes a + list of schema to get functions/procedures meta information that + are used in the current schema export. When replacing call to + function with OUT or INOUT parameters, if a function is declared + in an other package then the function call rewriting can not be + done because Ora2Pg only knows about functions declared in the + current schema. By setting a comma separated list of schema as + value of the directive, Ora2Pg will look forward in these packages + for all functions, procedures and packages declaration before + proceeding to current schema export. + - Add PG_SUPPORTS_NAMED_OPERATOR to control the replacement of the + PL/SQL operator used in named parameter => with the PostgreSQL + proprietary operator := Disable this directive if you are using + PG < 9.5 + - Add a warning when Ora2Pg reorder the parameters of a function + following the PostgreSQL rule that all input parameters following + a parameter with a default value must have default values as well. + In this case, Ora2Pg extracts all parameters with default values + and put them at end of the parameter list. This is to warn you + that a manual rewrite is required on calls to this function. + +New command line options have been added: + + - Add -N | --pg_schema command line option to be able to override + the PG_SCHEMA configuration directive. When this option is set + at command line, EXPORT_SCHEMA is automatically activated. + - Add --no_header option with equivalent NO_HEADER configuration + directive to output the Ora2Pg header but just the translated + code. + +There is also some behavior changes from previous release: + + - Remove SysTimestamp() from the list of not translated function, + it is replaced with CURRENT_TIMESTAMP for a long time now. + - Change migration assessment cost to 84 units (1 day) for type + TABLE, INDEX and SYNONYM and to 168 units (2 days) for TABLE + PARTITION and GLOBAL TEMPORARY TABLE, this is more realistic. + - Set minimum assessment unit to 1 when an object exists. + Improve PL/SQL code translation speed. + - Change behavior of COMPILE_SCHEMA directive used to force Oracle + to compile schema before exporting code. When this directive is + enabled and SCHEMA is set to a specific schema name, only invalid + objects in this schema will be recompiled. When SCHEMA is not set + then all schema will be recompiled. To force recompile invalid + object in a specific schema, set COMPILE_SCHEMA to the schema name + you want to recompile. This will ask to Oracle to validate the + PL/SQL that could have been invalidate after a export/import for + example. The 'VALID' or 'INVALID' status applies to functions, + procedures, packages and user defined types. + - Default transaction isolation level is now set to READ COMMITTED + for all action excluding data export. + - Oracle doesn't allow the use of lookahead expression but you may + want to exclude some objects that match the ALLOW regexp you have + defined. For example if you want to export all table starting + with E but not those starting with EXP it is not possible to do + that in a single expression. + Now you can start a regular expression with the ! character to + exclude all objects matching the regexp given just after. Our + previous example can be written as follow: ALLOW E.* !EXP.* + it will be translated into + + REGEXP_LIKE(..., '^E.*$') AND NOT REGEXP_LIKE(..., '^EXP.*$') + + in the object search expression. + - Fix quoting of PG_SCHEMA with multiple schema in search path. The + definition of the search path now follow the following behavior: + * when PG_SCHEMA is define, always set search_path to its value. + * when EXPORT_SCHEMA is enabled and SCHEMA is set, the search_path + is set the name of the schema. + - Remove forcing of export_schema when pg_schema is set at command + line. This could change the behavior of some previous use of these + variables and the resulting value of the search_path but it seems + much better understandable. + - Rewrite translation of raise_application_error to use RAISE + EXCEPTION with a message and the SQLSTATE code. Oracle user + defined code -20000 to -20999 are translated to PostgreSQL + user define code from 45000 to 45999. Call to + raise_application_error(mySQLCODE, myErrmsg); + will be translated into + RAISE EXCEPTION '%', myErrmsg USING ERRCODE = mySQLCODE; + - Remove migration assessment cost for TG_OP and NOT_FOUND they + might be fully covered now. + +Here is the complete list of changes: + + - Fix bad inversion of HAVING/GROUP BY clauses. Thanks to bhoot929 + for the report. + - Fix case of non translation of type in CAST() function. Thanks to + Keshavkumbham for the report. + - Fix spatial data export when using direct import into PostgreSQL + and WKT or INTERNAL format. This can still be improved. Thanks to + Valeria El-Samra for the report. + - Improve translation of trunc() into date_trunc. Thanks to bhoot929 + for the report. + - Translate to_char() without format into a simple cast to varchar. + Thanks to bhoot929 for the report. + - Fix line comment which does not disable multi-line comment. + Thanks to Pavel Stehule for the report. + - Fix overridden of output file global_variables.conf with + multiple packages. Thanks to Oliver Del Rosario for the report. + - Fix export of data stored in a nested user defined type. Thanks + to lupynos for the report. + - Fix data export from Oracle user defined types, where output of + ROW(...) does not distinguish numeric from string or other types + that need to be formatted. Thanks to Petr Silhak for the report. + - Fix broken replacement of package procedure name. Thanks to Pavel + Stehule for the report. + - Add FLOWS_010600 to the objects exclusion listr. + - Improve view/trigger migration assessment accuracy. + - Fix OUTER JOIN (+) translation, all join filters with constant + was written into the WHERE clause by default. Write them into the + JOIN clause. + - Fix weight of the number of triggers and views in the report with + a limit of 2 man-days, of course SQL difficulties are still add + after this limit. + - Fix alias added to condition which is not a sub query. Thanks to + nitinmverma for the report. + - Fix wrong translation of OUTER JOIN with subquery in FROM clause. + Thanks to nitinmverma for the report. + - Fix typo preventing exclusion of system SYNONYM. + - Fix an other case of bad translation of END fct_name. Thanks to + nitinmverma for the report. + - Fix unwanted translation of REGEXP_SUBSTR() in REGEXP_SUBSTRING(). + Thanks to nitinmverma for the report. + - Fix broken translation of decode(). Thanks to nitinmverma for the + report. + - Fix error "Malformed UTF-8 character (unexpected continuation byte + 0xbf, with no preceding start byte) in pattern match" by including + an arbitrary non-byte character into the pattern. Thanks to Bob + Sislow for the report. + - Fix missing translation of trunc() with date_trunc(). Thanks to + nitinmverma for the report. + - Add migration assessment weight to concatenation. + - Fix space between operator, ex: a < = 15 must be translated as + a <= 15. Thanks to nitinmverma for the report. + - Handle translation of named parameters in function calls. Thanks + to Julien Rouhaud for the patch. + - Fix missing renaming of _initial_command() method. + - Fix right outer join translation by converting them to left outer + join first. Thanks to Julien Rouhaud for the hint. + - Fix TEST on default value count and functions belonging to others + than public schema, especially functions of packages. + - Fix default number of man-days in migration assessment. Thanks to + Nate Fitzgerald for the report. + - Add host information into the filename of the report to prevent + some additional case of report overriding. Thanks to Aurelien + Robin for the report. + - Prevent ora2pg script to complain if no ora2pg.conf file is found + when a DSN is passed at command line and that user connection is + set in the environment variables. + - Do not declare a function stable if there is update/insert/delete + statement inside. + - Improve ora2pg_scanner to generate report of each instance schema + when the schema to audit is not set. Thanks to Thomas Reiss for + the patch. + - Fix parser failure with quote in comments. Thanks to Eric Delanoe + for the report. + - Fix case where NVL is not replaced by COALESCE. + - Add parenthesis to simple package.function call without parameter. + - Fix replacement of INSTR() with optional parameters. Thanks to + Pavel Stehule for the report. + - Translate SQL%ROWCOUNT to GET DIAGNOSTICS rowcount = ROW_COUNT. + Thanks to Pavel Stehule for the report. + - Add translation of SQL%FOUND. Thanks to Pavel Stehule. + - Remove qualifier in create type, "CREATE TYPE fusion.mytable AS + (fusion.mytable fusion.finalrecord[]);" becomes "CREATE TYPE + fusion.mytable AS (mytable fusion.finalrecord[]);". Thanks to + Julien Rouhaud for the report. + - Fix extra comma in FROM clause of triggers in outer join + translation. Thanks to Pavel Stehule fora the report. + - Use record for out parameters replacement only where there is more + than one out parameter. Thanks to Pavel Stehule for the patch. + - Add date type to the inout type array. Thanks to Pavel Stehule for + the report. + - Remove possible last spaces in inout type detection. + - Fix REGEXP_LIKE translation. + - Fix count of default values during test action. + - Fix removing of double quote over column name in view declaration. + - Do not set default value if it is NULL, this is already the case. + - Fix data export that was truncated to the last DATA_LIMIT lines. + Thanks to Michael Vitale for the report. + - Fix an other bug in rewriting call to function with OUT parameter. + Thanks to Pavel Stehule for the report. + - Fix autodetection of composite out parameters. + - Merge typo on PLPGSQL + - Fix typo to PLPGSQL keyword. Thanks to Vinayak Pokale. + - Fix regexp failure in is_reserved_words() method. Thanks to + Patrick Hajek for the patch. + - Use only one declaration of ora2pg_r RECORD, it is reusable. + - Fix transformation of procedure CALL with OUT parameter that can't + works when procedure/function has minimally one OUT parameter is + of composite type. Thanks to Pavel Stehule for the patch. + - Second attempt to fix outer join translation in triggers. Thanks + to Pavel Stehule for precious help. + - Fix RAISE NOTICE replacement with double % placeholder. Thanks to + Pavel Stehule for the report. + - Fix call to function replacement for function registered with + double quote. Thanks to Pavel Stehule for the report. + - Change assessment score of TG_OP. + - Fix replacement of outer join in triggers by adding pseudo tables + NEW and OLD to the list of tables. Thanks to Pavel Stehule for the + report. + - Handle custom exception in declare section of triggers. + - Fix FUNCTION_CHECK option, it will be set in all file header. + Thanks to Pavel Stehule for the report. + - Replace call to ALL_TABLES to USER_TABLES when USER_GRANTS is + enabled. Thanks to Bob Sislow. + - Removed PRAGMA EXCEPTION_INIT() from declare section. Thanks to + Pavel Stehule for the report. + - Fix constant string that was breaking the parser. Thanks to Pavel + Stehule for the report. + - Fix missing space between EXCEPTION and WHEN. Thanks to Pavel + Stehule for the report. + - Fix broken function header resulting in missing space before OUT + keyword. Thanks to Pave Stehule for the report. + - Fix invalid RAISE command. Thanks to Pavel Stehule for the report. + - Add port information to ORACLE_DSN in documentation and config + file. Thanks to Bob Sislow for the report. + - Fix broken declaration in triggers related to FOR cycle control + variable when there is no declare section. Thanks to Pavel + Stehule for the report. + - Fix broken declaration in triggers related to FOR cycle control + variable. Thanks to Pavel Stehule for the report. + - Fix unterminated C style comment in trigger. Thanks to Pavel + Stehule for the report. + - Fix bug in package+function precedence replacement. Thanks to + Eric Delanoe for the report. + - Fix unwanted and broken export of tables created with CREATE TABLE + tablename OF objType. Thanks to threenotrump for the report. + - Add explanation on using REPLACE_AS_BOOLEAN when REPLACE_TABLES or + REPLACE_COLS is also used on the same object. Thanks to Brendan + Le Ny for the report. + - Fix unwanted data export of materialized view. Thanks to Michael + Vitale for the report. Fix ORA-00942 with table that are not yet + physically created and has no data. + - Fix calling functions with same name declared in several packages. + The one declared in current package now takes precedence. Thanks + to Eric Delanoe for the report. + - Change zero-length lob/long to undef workaround for a bug in + DBD::Oracle with the ora_piece_lob option (used when no_lob_locator + is enabled) where null values fetch as empty string for certain + types. Thanks to Alice Maz for the patch. + - Fix createPoint() spatial method issue. Thanks to jwl920919. + - Fix comment on REPLACE_COLS directive. + - Fix an other issue in transformation of TYPE x IS REF CURSOR. + Thanks to Pavel Stehule for the report. + - Fix an other case of broken declaration in triggers related to FOR + cycle control variables. Thanks to Pavel Stehule for the report. + - Fix broken declaration in triggers related to FOR cycle control + variables with empty DECLARE section. + - Fix other case of replacement by EXIT WHEN NOT FOUND. + - Fix output of global_variables.conf file when OUTPUT_DIR is not + set. Fix non replacement of global variables. + - Remove some AUTHID garbage in procedure declaration generated by + a migration. + - Fix trigger name quoting with non supported character. Thanks to + Michael Vitale for the report. + - Fix use of nextval() in default value. + - Fix alias append in from clause of the extract() function. Thanks + to Narayanamoorthys for the report. + - Disable direct export to partition for PostgreSQL 10 if directive + PG_SUPPORTS_PARTITION is enabled, import must be done in the main + table. + - Do not export partitions of a materialized view. Thanks to Michael + Vitale for the report. + - Fix wrong replacement of keyword after END. Thanks to Pavel + Stehule for the report. + - Remove Oracle hints from queries, they are not supported and can + break comments. Thanks to Pavel Stehule for the report. + - Fix unstable transformation of TYPE x IS REF CURSOR. Thanks to + Pavel Stehule for the report. + - Fix data export failure when no table match the ALLOW/EXCLUDE + filter. Thanks to threenotrump for the report. + - Add missing search_path to FKEY dedicated file. Thanks to Michael + Vitale for the report. + - Apply default oject name exclusion to synonym. + - Skip some PL/SQL translation in migration assessment mode. + - Change default type for virtual column whit round() function. + Thanks to Julien Rouhaud for the report. + - Fix bug with EXIT WHEN command. Thanks to Pavel Stehule. + - Fix an other wrong replacement of DECODE in UPDATE statement. + Thanks to Pavel Stehule for the report. + - Fix wrong replacement of DECODE. Thanks to Pavel Stehule. + - Fix unwanted replacement of INTO STRICT when this is an INSERT + statement. Thanks to Pavel Stehule for the report. + - Fix potential regexp error with special character in outer join + filters. Thanks to Adrian Boangiu for the report. + - Fix parsing of PK from file. Thanks to Julien Rouhaud. + - Fix parsing of FK from file. Thanks to Julien Rouhaud. + - Fix count of unique and primary key in TEST export. Thanks to + Liem for the report. + - Fix reserved keyword rewrite using double quote when directive + USE_RESERVED_WORDS is enabled. Thanks to Michael Vitale. + - Remove EVALUATION CONTEXT object type from migration assessment. + - Add QUEST_SL_% pattern to the list of table that must be excluded + from export. + - Fix data export when BFILE are translated as text. Thanks to + Michael Vitale for the report. + - Fix export of package when a comment is placed just before the + AS/IS keyword. Thanks to Michael Vitale for the report. + - Fix other cases of function call replacement when they are + declared in different packages and one with OUT parameter and + the other one with only IN parameters. Thanks to Eric Delanoe. + - Fix inconsistent behavior of import_all script with -h and -d + Under Linux: When -h not specified, script defaults to unix domain + sockets for psql and localhost for perl (which may error depending + on pg_hba.conf). Now defaults to more performing sockets. -d + wasn't passing DB name to some psql calls where it's necessary. + Thanks to BracketDevs for the patch. + - Fix file handle close when compression is enabled. Thanks to + Sebastian Albert for the report. + - Add information about ora2pg behavior during data export to files + when files already exists. Thanks to Michael Vitale. + - Update readme to provide tar command for bzip2 file. Thanks to + Tom Pollard for the patch + - Fix unwanted FTS_INDEXES empty file and append unaccent extension + creation if not exists. Thanks to Michael Vitale for the report. + - Fix missing explicitly declared variable for cycle in trigger. + Thanks to Pavel Stehule for the report. + - Fix global type/cursor declaration doubled in package export. + - Always translate Oracle SELECT ... INTO to SELECT ... INTO STRICT + in plpgsql as Oracle seems to always throw an exception. Thanks + to Pavel Stehule for the report. + - Fix too much semicolons on end of function. Thanks to Pavel + Stehule for the report. + - Fix ALTER TABLE to set the owner when table is a foreign table. + Thanks to Narayanamoorthys for the report. + - Fix case of untranslated procedure call when there was parenthesis + in the parameters clause. Thanks to Pavel Stehule for the report. + - Fix broken variable declaration with name containing DEFAULT. + Thanks to Pavel Stehule for the report. + - Change query ORDER BY clause to view export query. + - Fix missing replacement of quote_reserved_words function by new + quote_object_name. Thanks to Sebastian Albert for the report. + - Fix REPLACE_COLS replacement of column name in UNIQUE constraint + definition. Thanks to Bernard Bielecki for the report. + - Fix export of Oracle unlogged table that was exported as normal + tables. + - Fix regression in package function calls rewrite leading to append + unwanted comma when replacing out parameters. Thanks to Pavel + Stehule and Eric Delanoe for the report. + - Fix removing of function name after END keyword. Thanks to Pavel + Stehule for the report. + - Fix bug in package function extraction. + - Improve VIEW export by only looking for package function name and + fix a bug that was including unwanted "system" package definition. + Also fix a potential bad rewriting of function call. Thanks to + Eric Delanoe for the report. + - Fix an other case of missing PERFORM replacement. Thanks to Pavel + Stehule for the report. + - Fix remplacement of "EXIT WHEN cursor%NOTFOUND". Thanks to Pavel + Stehule for the report. + - Fix missing conversion of type in cast function. Thanks to Michael + Vitale for the report. + - Fix TO_NUMBER that is now translated as a cast to numeric to + correspond to the default behavior in Oracle. Thanks to Pavel + Stehule for the report. + - Fix replacement of function call from different schema, especially + in overloaded cases. + - Remove OUT parameter from the argument list of function call. + Thanks to Pavel Stehule for the report. + - Fix wrong replacement in FOR ... IN loop inserting EXCLUDE in the + statement. Thanks to Pavel Stehule for the report. + - Translate Oracle proprietary VALUES without parenthesis with the + proprietary syntax of POstgreSQL. Thanks to Pavel Stehule for the + report. + - Fix function header translation when a comment is present between + closing parenthesis and the IS keyword. Thanks to Pavel Stehule + for the report. + - Fix RETURNS in autonomous transaction call when there is OUT + parameters. Thanks to Pavel Stehule for the report. + - Fix call to BMS_UTILITY.compile_schema() when COMPILE_SCHEMA is + enable. Thanks to PAvel Stehule for the report. + - Fix export of function and procedure with same name in different + schema. Thanks to Pavel Stehule for the report. + - Fix detection and replacement of global variable in package that + was producing invalid code export. Fix progress bar on procedure + export. + - Fix regression in global variables default value export. + - Rewrite multiprocess for procedure and function export to solve + some performances issues. + - Do not waste time trying to replace function call when it is not + found in the current code. + - Fix default value for FILE_PER_FUNCTION when parallel mode is + enabled. + - Output a fatal error with export type TABLE and multiple schema set + to PG_SCHEMA when EXPORT_SCHEMA is enabled. + - Fix replacement of function name with package prefix. + - Fix documentation of PG_SCHEMA directive, especially on the use of + a schema list. Thanks to Michael Vitale for the report. + - Fix translation of INSTR() function. + - Improve speed in function translation by not calling twice + Ora2Pg::PLSQL::convert_plsql_code() on declare and code section. + Thanks to Pavel Stehule for the profiling. + - Fix unwanted replacement of SYSDATE, SYSTIMESTAMP and some other + when they are part of variable or object name. Add rewrite of + REF CURSOR during type translation. + - Require support of LATERAL keyword for DATADIFF (Pg >= 9.3). + Patch from Sebastian Albert. + - Do not call replace_sdo_function(), replace_sdo_operator() and + replace_sys_context() if the string SDO_ or SYSCONTEXT is not + found. This might save some performances. + - Remove the RETURNS clause when there is an OUT parameter + PostgreSQL choose correct type by self. Thanks to Pavel Stehule + for the report. + - Add a note about performance improvement by updating stats on + Oracle. Thanks to Michael Vitale for the report. + - Remove newline characters in REVOKE statement when embedded in + a comment. Thanks to Pavel Stehule for the report. + - Fix replacement with PERFORM into package extracted from an + Oracle database. Thanks to Eric Delanoe for the report. + - Fix translation of call to function with out parameters. + Thanks to Pavel Stehule for the report. + - Fix case where call to procedure without parameter was not + prefixed by PERFORM or when called in a exception statement. + Thanks to Eric Delanoe for the report. + - Add function quote_object_name to handle all cases where object + name need to be double quoted (PRESERVE_CASE to 1, PostgreSQL + keyword, digit in front or digit only and non supported character. + Thanks to liemdt1811 for the report. + - Add a note about RAW(n) column with "SYS_GUID()" as default value + that is automatically translated to type of the column 'uuid' + by Ora2Pg. + - Remove old column count check to use char_length. Thanks to + Alice Maz for the patch. + - Fix some raise_application_error that was not replaced with a + global rewrite of remove comments and text constants to solve + some other issues like rewriting of package function call in + dynamic queries. Thanks to PAvel Stehule for the report. + - Fix cycle variable not generated for LOOP IN SELECT in trigger. + Thanks to Pavel Stehule for the report. + - Fix procedures with OUT parameters not processed in triggers. + Thanks to Pavel Stehule for the report. + - Remove other case where PERFORM must be or must not be inserted. + - Remove case where PERFORM can be inserted. Thanks to Pavel + Stehule and Eric Delanoe for the report. + - Fix missing ; in some raise_application_error translation. Thanks + to Pavel Stehule for the report. + - Fix missing PERFORM in front of direct call to function and the + rewrite of direct call to function with out parameters. Thanks + to Eric Delanoe for the report. + - Fix translation of rownum when the value is not a number. Thanks + to Pavel Stehule for the report. + - Fix missing space between cast and AS keyword. Thanks to Pavel + Stehule for the report. + - Fix translation of views and add support to comment inside views. + Thanks to Pavel Stehule for the report. + - Fix removing of AS after END keyword. Thanks to Pavel Stehule for + the report. + - Fix type in CAST clause not translated to PostgreSQL type. Thanks + to Pavel Stehule for the report. + - Treat citext type as text. Thanks to Tomasz Wrobel for the patch. + - Fix packages migration assessment that was broken with parser + rewriting on package extraction. + - Rewrite parsing of PL/SQL packages to better handle package + specification and especially types and global variables from this + section. + - Fix raise_application_error translation by removing extra boolean + parameter. + - Improve comments processing. + - Fix package function name replacement adding a dot before package + name. Thanks to Eric Delanoe for the report. + - Add collect of functions/procedures metadata when reading DDL + from file. + - Fix replacement of function prefixed with their schema name. + Thanks to Eric Delanoe for the report. + - Try to minimized comment placeholders by aggregating multiline + comments. + - Remove new line character from _get_version() output. + - Fix ENABLE_MICROSECOND test condition on NLS_TIMESTAMP_TZ_FORMAT + setting. Thanks to Didier Sterbecq for the report. + - Fix another issue with Oracle 8i and table size extraction. + - Fix query to show column information on Oracle 8i + - Fix query to look for virtual column on Oracle 8i + - Fix query to list all table by size on Oracle 8i + - Prevent ora2pg to look for external table definition in Oracle 8i. + - Fix a regression on timestamp format setting for Oracle 8i. + - Fix some regression on queries with Oracle 8i. Thanks to Didier + Sterbecq for the report. + - Add a function to collect metadata of all functions. + - Don't create empty partition index file when there's no partition. + - Fix wrong translation in OPEN ... FOR statement. Thanks to Eric + Delanoe for the report. + - Fix call of method close() on an undefined value. Thanks to Eric + Delanoe for the report. + - Fix partition data export issues introduced with previous patches. + - Fix unterminated IF / ELSIF block in subpartition export. + - Fix subpartition export. Thanks to Maurizio De Giorgi for the + report. + - Force DATA_LIMIT default value to 2000 on Windows OS instead of + 10000 to try to prevent constant OOM error. Thanks to Eric Delanoe + for the report. + - Fix default partition table that was not used PREFIX_PARTITION. + Thanks to ssayyadi for the report. + - Limit datetime microsecond format to micro second (.FF6) as the + format can be FF[0..9] and PostgreSQL just have FF[0..6] + - Add to_timestamp_tz Oracle function translation. Thanks to Eric + Delanoe for the feature request. + - Fix custom data type replacement in function code. Thanks to Pavel + Stehule for the report. + - Fix non working INPUT_FILE configuration directive when action is + QUERY. Thanks to Eric Delanoe for the report. + - Fix unwanted global variable implicit declaration to handle + autonomous transaction parameters. Thanks to Eric Delanoe for the + report. + - Fix call to dblink in function with PRAGMA AUTONOMOUS_TRANSACTION + and no arguments. Thanks to Eric Delanoe for the report. + - Fix package constant translation. Thanks to Eric Delanoe. + - Fix unwanted alias on join syntax. Thanks to Eric Delanoe + for the report. + - Fix regression on dbms_output.put* translation. Thanks to Eric + Delanoe for the report. + - Fix handling of comments in statements to try to preserve them at + maximum in the outer join rewriting. + - Do not declare variable when it is an implicit range cursor, it + do not need to be declared. + - Export implicit variable in FOR ... IN ... LOOP as an integer if + it don't use a select statement and export it as a RECORD when a + statement is found. Thanks to Eric Delanoe and Pavel Stehule for + the report. + - Reduce migration assessment weight for CONNECT BY. + - Fix derived table pasted two times in from clause. Thanks to Pavel + Stehule for the report. + - Fix some other unexpected ";" in function code. Thanks to Pavel + Stehule for the report. + - Remove %ROWTYPE in return type of function. Thanks to Pavel + Stehule for the report. + - Fix doubled AND in expression when a parenthesis is in front after + rewriting. Thanks to Eric Delanoe for the report. + - Fix unexpected ";" in function after post-body END when a comment + is present. Thanks to Eric Delanoe for the report. + - Fix unexpected ";" in some function variable declaration when a + comment is at end of the declare section. Thanks to Eric Delanoe + for the report. + - Remove %ROWTYPE in function that have not been replaced with RECORD + for cursor declaration. Thanks to Eric Delanoe for the report. + - Fix removing of WHEN keyword after END. Thanks to Pavel Stehule for + the report. + - Fix missing table name with alias in from clause due to comments in + the clause. I have also merge right and left outer join translation + function into a single one, most of the code was the same. + - Fix output order of outer join. Thanks to Pavel Stehule for the + report. + - Fix untranslated outer join in nested sub query. Thanks to Pavel + Stehule for the report. + - Rewrite again the decode() translation as a single function call + for all replacement before any other translation. + - Append table filter to check constraints extraction. Thanks to + danghb for the report. + - Fix issue with parenthesis around outer join clause. Thanks to + Pavel Stehule for the report. + - Move remove_text_constant_part() and restore_text_constant_part() + function into the main module. + - Include decode() replacement in recursive function call. Thanks + to Pavel Stehule for the report. + - Prevent removing of parenthesis on a sub select. Thanks to Pavel + Stehule for the report. + - Fix missing table exclusion/inclusion in column constraint export. + Thanks to danghb for the report. + - Fix an alias issue in view parsed from file. + - Fix parsing of view from file when no semi comma is found. + - Remove FROM clause without alias from migration assessment. + - Fix order of outer join during translation. Thanks to Pavel + Stehule for the report. + - Fix case of missing alias on subquery in FROM clause. Thanks to + Pavel Stehule for the report. + - Fix missing alias replacement in nested subqueries. Thanks to + Pavel Stehule for the report. + - Fix wrong addition of aliases to using() in join clause + - Fix nested decode replacement producing invalid CASE expression. + Thanks to Pavel Stehule for the report. + - Append aliases to subqueries in the from clause that do not have + one. Thanks to Pavel Stehule for the report. + +2017 02 17 - v18.1 + +This release fix several issues reported on outer join translation +thanks to the help of Pavel Stehule and reapply the commit on virtual +column export that was accidentally removed from v18.0. It also adds +several new features: + + - Remove CHECK constraints for columns converted into boolean using + REPLACE_AS_BOOLEAN column. + - Oracle function are now marked as stable by default as they can + not modify data. + +Two new configuration directives have been added: + + - DATE_FUNCTION_REWRITE: by default Ora2pg rewrite add_month(), + add_year() and date_trunc() functions set it to 0 to force Ora2Pg + to not translate those functions if translated code is broken. + - GRANT_OBJECT: when exporting GRANT you can now specify a comma + separated list of objects which privileges must be exported. + Default is to export privileges for all objects. For example + set it to TABLE if you just want to export privilege on tables. + +and a new command line option: + + - Add -g | --grant_object command line option to ora2pg to be able + to extract privilege from the given object type. See possible values + with GRANT_OBJECT configuration directive. + +Here is the complete list of changes: + + - Remove empty output.sql file in current directory with direct data + import. Thanks to kuzmaka for the report. + - Fix shell replacement of $$ in function definition in Makefile.PL + embedded configuration file. Thanks to kuzmaka for the report. + - Fix shell replacement of backslash in Makefile.PL embedded + configuration file. Thanks to kuzmaka for the report. + - Add warning level to virtual column notice. + - Fix comment in where clause breaking the outer join association. + Thanks to Pavel Stehule for the report. + - Add parsing and support of virtual column from DDL file. + - Reapply commit on virtual column export that was accidentally + removed in commit d5866c9. Thanks to Alexey for the report. + - Fix mix of inner join and outer join not translated correctly. + Thanks to Pavel Stehule for the help to solve this issue. + - Fix additional comma in column DEFAULT value from DDL input file. + Thanks to Cynthia Shang for the report. + - Fix comments inside FROM clause breaking translation to ANSI outer + joins. Thanks to Pavel Stehule for the report. + - Fix replacement of sdo_geometry type into function. Thanks to + Saber Chaabane for the report. + - Fix subquery in outer join clause. Thanks to Saber Chaabane for + the report. + - Fix duplicated subqueries placeholder in the from clause. + Thanks to Saber Chaabane for the report. + - Fix replacement of subquery place older during outer join rewrite. + Thanks to Saber Chaabane for the report. + - Add DATE_FUNCTION_REWRITE configuration directive. By default + Ora2pg rewrite add_month(), add_year() and date_trunc() functions + set it to 0 to force Ora2Pg to not translate those functions if + translated code is broken. Thanks to Pavel Stehule for the feature + request. + - Do not report error when -g is used but action is not GRANT. + Thanks to Shane Jimmerson for the report. + - Oracle function can not modify data, only procedure can do that, + so mark them as stable. Thanks to Pavel Stehule for the report. + - Missed some obvious combination like upper/lower case or no space + after AND/OR on outer join parsing and some other issues. + - Add missing call to extract_subqueries() recursively. Thanks to + Pavel Stehule for the report. + - Add full support of outer join translation in sub queries. + - Add translation of mixed inner join and Oracle outer join. Thanks + to Pavel Stehule for the report. + - Fix missing space between keyword AS and END from the decode() + transformation. Thanks to Pavel Stehule for the report. + - Fix parsing of outer join with UNION and translation to left join. + Thanks to Pavel Stehule for the report. + - Remove CHECK constraints for columns converted into boolean using + REPLACE_AS_BOOLEAN column. Thanks to Shane Jimmerson for the + feature request. + - Fix regression on SQL and PLSQL rewrite when a text constant + contained a semi-comma. + - Add the GRANT_OBJECT configuration directive. When exporting GRANT + you can specify a comma separated list of objects for which the + privileges will be exported. Default is export for all objects. + Here are the possibles values TABLE, VIEW, MATERIALIZED VIEW, + SEQUENCE, PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM and + DIRECTORY. Only one object type is allowed at a time. For example + set it to TABLE if you just want to export privilege on tables. + You can use the -g option to overwrite it. + When used this directive prevent the export of users unless it is + set to USER. In this case only users definitions are exported. + - Add the -g | --grant_object command line option to ora2pg to be able + to extract privilege from the given object type. See possible values + with GRANT_OBJECT configuration directive. + - Improve replacement of ROWNUM by LIMIT+OFFSET clause. + - Fix extra semi-colon at end of statement. + - Override ora2pg.spec with Devrim's one but with String::Random + removing as it is no more used. + +2017 01 29 - v18.0 + +This new major release adds several new useful features and lot of +improvements. + + * Automatic rewrite of simple form of (+) outer join Oracle's + syntax. This major feature makes Ora2Pg become the first free + tool that is able to rewrite automatically (+) outer join in + command line mode. This works with simple form of outer join + but this is a beginning. + * Add export of Oracle's virtual column using a real column and + a trigger. + * Allow conversion of RAW/CHAR/VARCHAR2 type with precision in + DATA_TYPE directive. Useful for example to transform all RAW(32) + or VARCHAR2(32) columns into PostgreSQL special type uuid. + * Add export NOT VALIDATED state from Oracle foreign keys and check + constraints into NOT VALID constraints in PostgreSQL. + * Replace call to SYS_GUID() with uuid_generate_v4() by default. + * Add "CREATE EXTENSION IF NOT EXISTS dblink;" before an autonomous + transaction or "CREATE EXTENSION IF NOT EXISTS pg_background;". + * Major rewrite of the way Ora2Pg parse PL/SQL to rewrite function + calls and other PL/SQL to plpgsql replacement. There should not + be any limitation in rewriting when a function contains a sub + query or an other function call inside his parameters. + * Refactoring of ora2pg to not requires any dependency other than + the Perl DBI module by default. All DBD drivers are now optionals + and ora2pg will expect an Oracle DDL file as input by default. + * Add export of Oracle's global variables defined in package. They + are exported as user defined custom variables and available in + a session. If the variable is a constant or have a default value + assigned at declaration, ora2pg will create a new file with the + declaration (global_variables.conf) to be included in the main + configuration file postgresql.conf file. + * Create full text search configuration when USE_UNACCENT directive + is enabled using the auto detected stemmer or the one defined in + FTS_CONFIG. For example: + CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); + ALTER TEXT SEARCH CONFIGURATION fr ALTER MAPPING FOR + hword, hword_part, word WITH unaccent, french_stem; + CREATE INDEX place_notes_cidx ON places + USING gin(to_tsvector('fr', place_notes)); + +Changes and incompatibilities from previous release: + + * FTS_INDEX_ONLY is now enabled by default because the addition of + a column is not always possible and not always necessary where a + simple function-based index is enough. + * Remove use to setweigth() on single column FTS based indexes. + * Change default behaviour of Ora2Pg in Full Text Search index + export. + +A new command line option and some configuration directive have +been added: + + * Option -D | --data_type to allow custom data type replacement + at command line like in configuration file with DATA_TYPE. + * UUID_FUNCTION to be able to redefined the function called to + replace SYS_GUID() Oracle function. Default to uuid_generate_v4. + * REWRITE_OUTER_JOIN to be able to disable the rewriting of Oracle + native syntax (+) into OUTER JOIN if rewritten code is broken. + * USE_UNACCENT and USE_LOWER_UNACCENT configuration directives to + use the unaccent extension with pg_trgm with the FTS indexes. + * FTS_INDEX_ONLY, by default Ora2Pg creates an extra tsvector column + with a dedicated triggers for FTS indexes. Enable this directive + if you just want a function-based index like: + CREATE INDEX ON t_document USING + gin(to_tsvector('pg_catalog.english', title)); + * FTS_CONFIG, use this directive to force the text search stemmer + used with the to_tsvector() function. Default is to auto detect + the Oracle FTS stemmer. For example, setting FTS_CONFIG to + pg_catalog.english or pg_catalog.french will override the auto + detected stemmer. + +There's also lot fixes of issues reported by users from the past two +months, here is the complete list of changes: + + - Fix return type in function with a single inout parameter and a + returned type. + - Prevent wrong rewrite of empty as null when a function is used. + Thanks to Pavel Stehule for the report. + - Add the UUID_FUNCTION configuration directive. By default Ora2Pg + will convert call to SYS_GUID() Oracle function with a call to + uuid_generate_v4 from uuid-ossp extension. You can redefined it + to use the gen_random_uuid function from pgcrypto extension by + changing the function name. Default to uuid_generate_v4. Thanks + to sjimmerson for the feature request. + - Add rewrite of queries with simple form of left outer join syntax + (+) into the ansi form. + - Add new command line option -D | --data_type to allow custom data + type replacement at command line like in configuration file with + DATA_TYPE. + - Fix type in ROWNUM replacement expression. Thanks to Pavel Stehule + for the report. + - Add replacement of SYS_GUID by uuid_generate_v4 and allow custom + rewriting of RAW type. Thanks to Nicolas Martin for the report. + - Fix missing WHERE clause in ROWNUM replacement with previous patch + thanks to Pavel Stehule for the report. + - Fix ROWNUM replacement when e sub select is used. Thanks to Pavel + Stehule for the report. + - Fix wrong syntax in index creation with DROP_INDEXES enabled. + Thanks to Pave Stehule for the report. + - Remove replacement of substr() by substring() as PostgreSQL have + the substr() function too. Thanks to Pavel Stehule for the report. + - Move LIMIT replacement for ROWNUM to the end of the query. Thanks + to Pavel Stehule for the report. + - Fix text default value between parenthesis in table declaration. + Thanks to Pavel Stehule for the report. + - Fix return type when a function have IN/OUT parameter. Thanks to + Pavel Stehule for the report. + - Mark uuid type to be exported as text. Thanks to sjimmerson for + the report. + - Add EXECUTE to open cursor with like "OPEN var1 FOR var2;". Thanks + to Pavel Stehule for the report. + - Fix replacement of local type ref cursor. Thanks to Pavel Stehule + for the report. + - Add EXECUTE keyword to OPEN CURSOR ... FOR with dynamic query. + Thanks to Pavel Stehule for the report. + - Fix case sensitivity issue in FOR .. IN variable declaration + replacement. Thanks to Pavel Stehule for the report. + - Fix wrong replacement of cast syntax ::. Thanks to Pavel Stehule + for the report. + - Reactivate numeric cast in call to round(value,n). + - Close main output data file at end of export. + - Add virtual column state in column info report, first stage to + export those columns as columns with associated trigger. + - Fix unwanted replacement of REGEXP_INSTR. Thanks to Bernard + Bielecki for the report. + - Allow rewrite of NUMBER(*, 0) into bigint or other type instead + numeric(38), just set DATA_TYPE to NUMBER(*\,0):bigint. Thanks to + kuzmaka for the feature request. + - Export partitions indexes into PARTITION_INDEXES_....sql separate + file named. Thanks to Nicolas Martin for the feature request. + - Fix fatal error when schema CTXSYS does not exists. Thanks to + Bernard Bielecki for the report. + - Fix missing text value replacement. Thanks to Bernard Bielecki + for the report. + - Fix type replacement in declare section when the keyword END was + present into a variable name. + - Export NOT VALIDATED Oracle foreign key and check constraint as + NOT VALID in PostgreSQL. Thanks to Alexey for the feature request. + - Add object matching of regex 'SYS_.*\$' to the default exclusion + list. + - Fix UTF8 output to file as the open pragma "use open ':utf8';" + doesn't works in a global context. binmode(':encoding(...)') is + used on each file descriptor for data output. + - Improve parsing of tables/indexes/constraints/tablespaces DDL from + file. + - Improve parsing of sequences DDL from file. + - Improve parsing of user defined types DDL from file. + - Export Oracle's TYPE REF CURSOR with a warning as not supported. + - Replace call to plsql_to_plpgsql() in Ora2Pg.pm by a call to new + function convert_plsql_code(). + - Move export of constraints after indexes to be able to use USING + index in constraint creation without error complaining that index + does not exists. + - Add "CREATE EXTENSION IF NOT EXISTS dblink;" before an autonomous + transaction or "CREATE EXTENSION IF NOT EXISTS pg_background;". + - Improve parsing of packages DDL from file. + - When a variable in "FOR varname IN" statement is not found in the + DECLARE bloc, Ora2Pg will automatically add the variable to this + bloc declared as a RECORD. Thanks to Pavel Stehule for the report. + - Major rewrite of the way Ora2Pg parse PL/SQL to rewrite function + calls and other PL/SQL to plpgsql replacement. There should not + be limitation in rewriting when a function contains a sub query + or an other function call inside his parameters. + - Fix unwanted SELECT to PERFORM transformation inside literal + strings. Thanks to Pavel Stehule for the report. + - Fix bug in DEFAULT value rewriting. Thanks to Pavel Stehule for + the report. + - Fix replacement of DBMS_OUTPUT.put_line with RAISE NOTICE. + - Reset global variable storage for each package. + - Improve comment parsing in packages and prevent possible infinite + loop in global variable replacement. + - Add the REWRITE_OUTER_JOIN configuration directive to be able to + disable the rewriting of Oracle native syntax (+) into OUTER JOIN + if it is broken. Default is to try to rewrite simple form of + right outer join for the moment. + - Export types and cursors declared as global objects in package + spec header into the main output file for package export. Types + and cursors declared into the package body are exported into the + output file of the first function declared in this package. + - Globals variables declared into the package spec header are now + identified and replaced into the package code with the call to + user defined custom variable. It works just like globals variables + declared into the package body. + - Add auto detection of Oracle FTS stemmer and disable FTS_CONFIG + configuration directive per default. When FTS_CONFIG is set its + value will overwrite the auto detected value. + - Create full text search configuration when USE_UNACCENT directive + is enabled using the auto detected stemmer or the one defined in + FTS_CONFIG. For example: + CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); + ALTER TEXT SEARCH CONFIGURATION fr ALTER MAPPING FOR + hword, hword_part, word WITH unaccent, french_stem; + CREATE INDEX place_notes_cidx ON places + USING gin(to_tsvector('fr', place_notes)); + - Remove CONTAINS(ABOUT()) from the migration assessment, there no + additional difficulty to CONTAINS rewrite. + - Add ANYDATA to the migration assessment keyword to detect. + - Allow conversion of CHAR/VARCHAR2 type with precision in DATA_TYPE + directive. For example it's possible to transform all VARCHAR2(32) + columns only into PostgreSQL special type uuid by setting: + DATA_TYPE VARCHAR2(32):uuid + Thanks to sjimmerson for the feature request. + - Update year in copyrights + - Fix creation of schema when CREATE_SCHEMA+PG_SCHEMA are defined. + - Fix renaming of temporary file when exporting partitions. + - Move MODIFY_TYPE to the type section + - Update documentation about globals variables. + - Add export of Oracle's global variables defined in package. They + are exported as user defined custom variables and available in + a session. Oracle variables assignment are exported as call to: + PERFORM set_config('pkgname.varname', value, false); + Use of these variable in the code is replaced by: + current_setting('pkgname.varname')::global_variables_type; + the variable type is extracted from the pacjkage definition. If + the variable is a constant or have a default value assigned at + declaration, ora2pg will create file global_variables.conf with + the definition to include in postgresql.conf file so that their + values will already be set at database connection. Note that the + value can always modified by the user so you can not have exactly + a constant. + - Fix migration assessment of view. + - Remove call to FROM SYS.DUAL, only FROM DUAL was replaced. + - Replace call to trim into btrim. + - Improve rewrite of DECODE when there is function call inside. + - Add function replace_right_outer_join() to rewrite Oracle (+) + right outer join. + - Improve view migration assessment. + - Create a FTS section in the configuration file dedicated to FTS + control. + - Add USE_UNACCENT and USE_LOWER_UNACCENT configuration directives + to use the unaccent extension with pg_trgm. + - Do not create FTS_INDEXES_* file when there is no Oracle Text + indexes. + - Update query test score when CONTAINS, SCORE, FUZZY, ABOUT, NEAR + keyword are found. + - Remove use to setweigth() on single column FTS based indexes. + Thanks to Adrien Nayrat for the report. + - Update documentation on FTS_INDEX_ONLY with full explanation on + the Ora2Pg transformation. + - Refactoring ora2pg to not requires any dependency other than the + Perl DBI module by default. All DBD drivers are now optionals and + ora2pg will expect to received an Oracle DDL file as input by + default. This makes easiest packaging or for any distribution that + can not build a package because of the DBD::Oracle requirement. + DBD::Oracle, DBD::MySQL and DBD::Pg are still required if you want + Ora2Pg to migrate your database "on-line" but they are optional + because Ora2Pg can also convert input DDL file, this is the + default now. Thanks to Gustavo Panizzo for the feature request and + the work on Debian packaging. + - Remove String::Random dependency in rpm spec file, it is no used + even if it was mentioned into a comment. + - Exclude internal Oracle Streams AQ JMS types from the export. + Thanks to Joanna Xu for the report. + - Fix some other spelling issues. Thanks to Gustavo Panizzo for the + patch. + - Fix some spelling errors. Thanks to Gustavo Panizzo for the patch. + - Revert patch 697f09d that was breaking encoding with input file + (-i). Thanks to Gary Evans for the report. + - Add two new configuration directive to control FTS settings, + FTS_INDEX_ONLY and FTS_CONFIG. + +2016 11 17 - v17.6 + +This release adds several new features: + + * Adds export of Oracle Text Indexes into FTS or pg_trgm + based indexes, + * Add export of indexes defined on materialized views + * Allow export of materialized views as foreign tables + when export type is FDW. + * Add replacement of trim() by btrim(). + +Two new configuration directives have been added: + + * USE_INDEX_OPCLASS: when value is set to 1, this will force + Ora2Pg to export all indexes defined on varchar2() and char() + columns using *_pattern_ops operators. If you set it to a value + greater than 1 it will only change indexes on columns where the + character limit is greater or equal than this value. + + * CONTEXT_AS_TRGM: when enabled it forces Ora2Pg to translate + Oracle Text indexes into PostgreSQL indexes using pg_trgm + extension. Default is to translate CONTEXT indexes into FTS + indexes and CTXCAT indexes using pg_trgm. Some time using + pg_trgm based indexes is enough. + +There's also some fixes of issues reported by users, here is the +complete list of changes: + + - Fixed non-use of custom temp_dir (-T). Thanks to Sebastian + Albert for the patch. + - Make export of FTS indexes from materialized view work as + for tables. + - Fix drop of indexes during export of data when DROP_INDEXES + is enabled. + - Remove double quote in function and procedure name from an input + file to avoid creating a file with double quote in its name. + - Fix export of unique index associated to a primary key. + - Move OPTION (key "yes") of FDW table before NOT NUL constraint + and default clause. + - Fix some encoding issue during data export into file. + - Rename FTS indexes prefix output file into FTS_INDEXES and + export CTXCAT Oracle indexes as GIN pg_trgm indexes instead of + FTS indexes. + - Add export of indexes of type CTXCAT as FTS indexes. + - Export triggers and update order for FTS indexes to separate file + prefixed with FTS_INDEXES. + - Exclude from export synonyms starting with a slash that correspond + to incomplete deleted synonyms. Thanks to Nouredine Mallem for the + report. + - Add export of indexes defined on materialized views. Thanks to + Nouredine Mallem for the report. + - Fix export of foreign key and FTS indexes when looking at dba_* + tables and multiple different schemas have the same fk or context + indexes definition. Thanks to Nouredine Mallemfor the patch. + - Fix export of CONTEXT or FULLTEXT Oracle index into PostgreSQL + FTS with trigger and initial update statement. + - Add configuration directive USE_INDEX_OPCLASS to force Ora2Pg to + export all indexes defined on varchar2() and char() columns using + those operators. A value greater than 1 will only change indexes + on columns where the character limit is greater or equal than + this value. + - Fix FDW export of mysql tables. Thanks to yafeishi for the report. + - Fix decode() rewrite. Thanks to Jean-Yves Julliot for the report. + - Fix regression introduced into the export of NUMBER to integer + like PG types. + - Show partition name in progress bar instead of main table name. + +2016 10 20 - v17.5 + +This is a maintenance release to fix several issues reported by users. +There is also some major improvement and new feature. + +There is a new configuration directive or change default behavior: + + * Fix export of CLOBs and NCLOB that was truncated to 64 Kb. + * PG_BACKGROUND : when enabled autonomous transactions will be + built using Robert Haas extension pg_background instead of dblink. + Default is to still used dblink as pg_background is available + only for PostgreSQL >= 9.5. + * All Perl I/O now use the open pragma instead of calling method + binmode(). This will force input and output to utf8 using the + Perl pragma: + use open ':encoding(utf8)'; + when configuration directive BINMODE is not set or NLS_LANG is + set to UTF8. + * Ora2Pg will now export empty lob as empty string instead of NULL + when the source column has NOT NULL constraint and that directive + EMPTY_LOB_NULL is not activated. + * Improve and fix progress bar especially when using JOBS/-J option. + * Allow LOAD action to apply all settings defined in the input file + on each opened session, this allow to use LOAD with export schema + enabled. If settings are not set in the input file encoding and + search_path is set from the ora2pg configuration settings. + * NUMBER(*,0) is now exported as numeric(38) as well as a NUMBER + with DATA_SCALE set to 0, no DATA_PRECISION and a DATA_LENGTH + of 22. The last correspond to Oracle type INTEGER or INT. + * Allow conversion of type with precision in DATA_TYPE directive. + For example it is possible to transform all NUMBER(12,2) only + into numeric(12,2) by escaping the comma. Example: + DATA_TYPE NUMBER(12\,2):numeric(12\,2);... + * Write data exported into temporary files (prefixed by tmp_) and + renamed them at end of the export to be able to detect incomplete + export and override it at next export. + * Add export of type created in package declaration. + * Export foreign key when the referenced table is not in the + same schema. + * Enabled by default PG_SUPPORTS_CHECKOPTION assuming that your Pg + destination database is at least a 9.4 version. + * Add 12 units to migration assessment report per table/column + conflicting with a reserved word in PostgreSQL to reflect the + need of code rewriting. + * Output a warning when a column has the same name than a system + column (xmin,xmax,ctid,etc.) + * Replace SYSDATE by a call to clock_timestamp() instead of a call + to LOCALTIMESTAMP in plpgsql code. + * Add missing documentation about DISABLE_PARTITION directive used + to not reproduce partitioning into PostgreSQL and only export + partitioned data into the main table. + * Show partition name in progress bar instead of main table name. + +Here is the complete list of other changes: + + - Fix broken parallel table export (-P option). + - Fix export of CLOBs and NCLOB that was truncated to 64Kb. Thanks + to Paul Mzko for the patch. + - Fix database handle in error report. + - Fix use of wrong database handle to set search_path. Thanks to + Paul Mzko for the report. + - Ora2pg doesn't export schema ForeignKey constraint when connected + as different DBA user. Thanks to Paul Mzko for the patch. + - Fix Perl I/O encoding using open pragma instead of calling method + binmode(). Thanks to Gary Evans for the report. + - Force input to utf8 using Perl pragma: use open ':encoding(utf8)'; + when BINMODE is not set or NLS_LANG is UTF8. + - Force ora2pg to export empty lob as empty string instead of NULL + when the source column has a NOT NULL constraint and directive + EMPTY_LOB_NULL is not activated. Thanks to Valeriy for the report. + - Fix missing CASCADE attribute on fkey creation during data export + when DROP_FKEY was enabled. Thanks to ilya makarov for the report. + - Fix issue on converting NUMBER(*,0) to numeric, should be ported + to numeric(38). Thanks to ilya makarov for the report. + - Correct query for ForeignKey export from oracle. Thanks to ilya + makarov for the patch. + - Fix schema change in direct import of data to PostgreSQL. + - Change query for foreign key extraction to keep the column order. + Thanks to ilya makarov for the report. + - Write data exported into temporary files (prefixed by tmp_) and + renamed them at end of the export to be able to detect incomplete + export and override it at next export. Thanks to Paul Mkzo for + the feature request. + - Fix infinite loop in blob extraction when error ORA-25408 occurs + during ora_lob_read() call. Thanks to Paul Mzko for the report. + - Fix order of columns in foreign keys definition. Thanks to ilya + makarov for the report. + - Fix export of partition by range on multicolumn. Thanks to Rupesh + Admane for the report. + - Update reserved keywords list. Thanks to Nicolas Gollet for the + report. + - Add ON DELETE NO ACTION on foreign key creation (DROP_FKEY) to + obtain the same output than during constraints export. + - Fix export of foreign key that was duplicating the columns in both + part, local and foreign. Thanks to ilya makarov for the report. + - Remove call to to_char(datecol, format) when exporting date and + timestamp. This formating is no more needed as we are now forcing + NLS_DATE_FORMAT and NLS_TIMESTAMP_FORMAT when opening a connection + to Oracle using: + ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS + and + ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS + This may result on some speed improvment during data export. + - Fix parsing of packages from input file. + - Add export of type created in package declaration. Thanks to + dezdechado for the report. + - Fix converting of procedures with out arguments. Thanks to + dezdechado for the report. + - Update documentation about project management. + - Fix replacement of = NULL by IS NULL in update statement. + Thanks to dezdechado for the report. + - Fix parsing of trigger from file that was broken and new line + removed. Thanks to dezdechado for the report. + - Fix erasing of quotes from text in triggers. Thanks to dezdechado + for the report. + - Fix "return new" on trigger function when there is exception. + Thanks to dezdechado for the report and solution. + - Fix conversion of INTEGER and INT into numeric(38) instead of + numeric without precision. Thanks to dezdechado for the report. + - Fix export of foreign key when the referenced table is not in the + same schema. Thanks to Juju for the report. + - Fix ddl create schema when EXPORT_SCHEMA and CREATE_SCHEMA are + enabled but no schema is specified. + - Fix export of NCHAR that was converted as char but was loosing its + length definition. Thanks to lgerlandsen for the report. + - Fix parsing of views using WITH statements. Thank to dezdechado + for the report. + - Fix case that breaks views/triggers definition when a semicolon + is encountered in a string value when reading definition from + file. Thanks to dezdechado for the report. + - Fix included/excluded of sequences when using ALLOW/EXCLUDE + directives. Thanks to Roman Sindelar for the report. + - prepare options modified with some escaping improvements. Thanks + to ioxgrey for the patch. + - It seems that for a NUMBER with a DATA_SCALE set to 0, no + DATA_PRECISION and a DATA_LENGTH of 22 in ALL_TAB_COLUMNS, Oracle + use a NUMBER(38) instead. This correspond to Oracle type INTEGER + or INT. I don't really understand the reason of this behavior, + why not just using a data length of 38? ALL_TAB_COLUMNS and Ora2Pg + reports a data length of 22, now Ora2Pg will report NUMBER(38) like + the resulting type of the DESC command. + + The following Oracle table: + + CREATE TABLE TEST_TABLE ( FIELD_1 INTEGER, FIELD_2 NUMBER ); + + will be exported as follow by Ora2Pg: + + [1] TABLE TEST_TABLE (owner: HR, 0 rows) + FIELD_1 : NUMBER(38) => numeric + FIELD_2 : NUMBER(22) => bigint + + Oracle data type INTEGER and INT will be exported as numeric by + default instead of an integer. + - Fix parsing of function/procedure from file with comments after + BEGIN statement. + - Remove DEFERRABLE INITIALLY DEFERRED from CHECK constraints when + parsed from file. Thanks to Felipe Lavoura for the report. + - Fix double parenthesis in index definition when parsing index + creation from file. Thanks to Felipe Lavoura for the report. + - Fix parsing of COMMENT from file. + - Fix undetected native Oracle type bug. Thanks to kvnema for the + report. + - Fix unwanted text formatting with bind value in INSERT action + with direct import to PostgreSQL. Thanks to Oleg for the report. + - Fix inversion of UPDATE_RULE and DELETE_RULE in foreign key + creation for MySQL export. Thanks to Sebastian Albert for the + report. + - Update documentation about DEFER_FKEY and DROP_FKEY to report + the new behavior. + - Remove call to SET CONSTRAINTS ALL DEFERRED with direct import. + - Fix use of NULL value in bind parameter that should be undefined + (INSERT export mode only). Thanks to Oleg barabaka for the report. + - Remove replacement of direct call to functions with PERFORM, there + is too much false positive. Thanks to dezdechado for the reports. + - Fix a typo in SYSDATE replacement. Thank to dezdechado for report. + - Remove rewrite of concatenation in RAISE EXCEPTION. Thanks to + dezdechado for the report. + - Fix replacement of raise_application_error() when first argument + is a variable. Thanks to dezdechado for the report. + - Fix wrong insertion of PERFORM before some function calls. Thanks + to dezdechado for the report. + - Replace SYSDATE by a call to clock_timestamp() instead of call to + LOCALTIMESTAMP in plpgsql code. Thanks to aleksaan for the report. + - Allow use of comma for object name list separator instead of space + as workaround on Window OS. + - Fix documentation about MODIFY_TYPE. Thanks to Nicolas Gollet for + the report. + - Add missing documentation about DISABLE_PARTITION directive used + to not reproduce partitioning into PostgreSQL and only export + partitioned data into the main table. Thanks to Nicolas Gollet + for the report. + - Add information about how to export LONG RAW data type. They need + to be exported as BLOB before into Oracle to be exported as BYTEA. + - Fix case where select was wrongly replaced by perform in INSERT + INTO with SELECT statement. Thanks to dezdechado for the report. + - Fix links to ora2pg presentation. Thanks to Daniel Lenski for the + patch. + - Fix input parameters after one with a default value must also have + defaults. Thanks to v.agapov fot the patch. + - Fix debug mode that was interromping the last running table dump. + Thanks to calbiston for the report. + +2016 04 21 - v17.4 + +Errata in first release attempt. + + - Fix previous patch that does not handle blob case but just clob + - Forgot to change back the query when EMPTY_LOB_NULL is not activated. + - Put parenthesis around AT TIME ZONE expression + +This is a maintenance release to fix several issues reported by users. +There is also some major data export speed improvement thanks to the +work of PostgreSQL Pro and a new RPM spec file provided by Devrim +Gunduz to be able to build RPM package for Ora2Pg. + +There is a new configuration directive: + + - EMPTY_LOB_NULL: when enabled force empty_clob() and empty_blob() + to be exported as NULL instead as empty string. + +Here is the complete list of other changes: + + - Add EMPTY_LOB_NULL directive to force empty_clob() and empty_blob() + to be exported as NULL instead as empty string. This might improve + data export speed if you have lot of empty lob. Thanks to Alex + Ignatov for the report. + - Fix import_all.sh script to import grant and tablespace separately + as postgres user and just after indexes and constraints creation. + - Add parsing of tablespace from "alter table ... add constraint" + with DDL input file. Thanks to Felipe Lavoura. + - Remove --single-transaction in import_all.sh script with TABLESPACE + import. Thanks to Guillaume Lelarge for the report. + - Fix Makefile.PL to used with latest spec file from Devrim Gunduz + and following the advice of calbiston. + - Update spec file to v17.6 and latest change to Makefile.PL + - Replace ora2pg.spec by postgressql.org spec file by Devrim Gunduz. + - Generate man page to avoids rpmbuild error. + - Fix Windows install. Thanks to Lorena Figueredo for the report. + - Remove "deferrability" call for mysql foreign keys. Thanks to + Jean-Eric Cuendet for the report. + - Fix issue in restoring foreign key for mysql data export. Thanks + to Jean-Eric Cuendet for the report. + - Remove connection test to PostgreSQL instance as postgres or any + superuser in import_all.sh + - Fix creation of configuration directory. + - Fix Makefile to dissociate CONFDIR and DOCDIR from PREFIX or + DESTDIR. Thanks to Stephane Schildknecht for the report. + - Fix date_trunc+add_month replacement issue. Thanks to Lorena + Figueredo for the report. + - Do not replace configuration directory in scripts/ora2pg if this + is a RPM build. Thanks to calbiston for the report. + - Return empty bytea when a LOB is empty and not NULL. + - Regular expressions and conditions checks improvement in method + format_data_type() to make it a bit faster on huge tables. Thanks + to Svetlana Shorina for the patch. + - Fix INSERT using on the fly data import with boolean values. + Thanks to jecuendet for the report. + - Allow MySQL data type to be converted into boolean. Thanks to + jecuendet for the report. + - Fix export of BIT mysql data type into bit bit varying. Thanks + to jecuendet for the report. + - Fix call to escape_copy/escape_insert function call. + +2016 03 26 - v17.3 + +This release fix two regressions introduced in latest release. + + * Fix major bug in data export. Thanks to Frederic Guiet for the report. + * Fix another regression with character data that was not escaped. Thanks + to Frederic Guiet for the report. + +2016 03 24 - v17.2 + +This is a maintenance release to fix several issues reported in new +LOB extraction method. There is also some feature improvement: + + * Allow NUMBER with precision to be replaced as boolean. + * Allow full relocation of Ora2Pg installation using for + example: perl Makefile.PL DESTDIR=/opt/ora2pg + +Here is the complete list of other changes: + + - Allow NUMBER with precision to be replaced as boolean. Thanks + to Silvan Auer for the report. + - Force empty LOB to be exported as NULL when NO_LOB_LOCATOR is + activated to have the same behavior. + - Fix case where a LOB is NULL and ora2pg reports error : + DBD::Oracle::db::ora_lob_read: locator is not of type OCILobLocatorPtr + LOB initialised with EMPTY_CLOB() are also exported as NULL + instead of \\x + - Fix replacement with PERFORM after MINUS. Thanks to Stephane + Tachoires for the report. + - Comment DBMS_OUTPUT.ENABLE calls. Thanks to Stephane Tachoire for + the report. + - Fix wrong replacement of SELECT by PERFORM after EXCEPT. Thanks + to Stephane Tachoire for the report. + - Apply ORACLE_COPIES automatic predicate on custom queries set with + REPLACE_QUERY if possible. Thanks to pawelbs for the report. + - Fix install of ora2pg.conf file in /etc/ instead of /etc/ora2pg/. + Thanks to pawelbs for the report. + - Add debug information before searching for custom type. + - Attempt to fix error "ORA-01002: fetch out of sequence" when exporting + data from a table with user defined types and ORACLE_COPIES. Thanks to + pawelbs and Alex Ignatov fir the report. + - Fix replacement of path to configuration file in scripts/ora2pg + - Remove report sample from documentation about migration assessment + report and replace it with a href link. Fix comment about export of + domain index. + - Always prefix table name with schema in Oracle row count, to prevent + failure when the schema is not the connexion default. + - Add pattern TOAD_PLAN_.* to the internal table exclusion list. + - Fix modification of database owner search_path in import_all.sh auto + generated script. Thanks to Stephane Tachoire for the report. + +2016 02 29 - v17.1 + +This is a maintenance release to fix several issues reported in new +TEST action. There is also some feature improvement: + + * Add OPTIONS (key 'true') on table FDW export when a column is + detected as a primary key. + * Add DELETE configuration directive that allow a similar feature + than the WHERE clause to replace TRUNCATE call by a "DELETE FROM + table WHERE condition". This feature can be useful with regular + "updates". Thanks to Sebastien Albert for the feature request. + +Here is the complete list of other changes: + + - Fix the counter of user defined types and sequences in TEST action + - Fix COPY import of data from column with user defined type with + NULL value. + - Fix DBD::Pg segmentation fault with direct INSERT import from + column with user defined type. + - Fix TEST action with multiple PG_SCHEMA export. Thanks to Michael + Vitale for the report. + - Fix documentation about PG_SCHEMA + +2016 02 22 - v17.0 + +This new major release adds a new action type TEST to obtain a count +of all objects at both sides, Oracle and PostgreSQL, to perform a +diff between the two database and verify that everything have been +well imported. It also fixes several issues reported by users. + +A new ora2pg command line option have been added to ora2pg script: + + * Add --count_rows command line option to perform a real row count + on both side, Oracle and PostgreSQL, in TEST report. + +Here is the complete list of changes and bugfixes: + + - Prefix direct call to function with a call to PERFORM. Thanks to + Michael Vitale for the feature request. + - Fix revoke call on function with multiline parameters declaration. + - Fix auto setting of internal schema variable with mysql. + - Define ORACLE_HOME with the corresponding environment variable in + generic configuration when available and --init_project is used. + Thanks to Stephane Tachoires for the report. + - Fix documentation about exporting view as table. + - Remove some obsolete code and display information when a view is + exported as table. + - Fix empty LOB data export with Oracle Lob locator (NO_LOB_LOCATOR + set to 0). + - Fix data export of partitions with single process mode and when + FILE_PER_TABLE is enabled. + - Fix export of RAW data type. + - Fix missing $ to call to self variable. Thanks to NTLIS and Sirko + for the report. + - Force FKey to be initially immediate when deferred is not set. + Thanks to Stephane Tachoire for the report. + - Fix count of check constraint when a schema is forced. + - Allow TEST action on mysql database too with some improvements + and bug fix on the feature. + - Fix index column renaming in mysql export. + - Fix dblink extraction query when an exclusion is set. + - Fix sequence name auto generation for mysql serial number. + - Add --count_rows command line option to make optional the real + row count in TEST report. This is useful when you have lot of + data and do not want to loose time in call to count(*). + - Update documentation about the TEST action and usage, see + chapter "Test the migration". + - Apply schema context on PostgreSQL side with TEST action. + - Add TEST action type to ask Ora2Pg to count rows and all objects + at both sides, Oracle and PostgreSQL, to verify that everything + have been well imported. + - Fix missing export of foreign keys on multiple columns, ex: + ALTER TABLE products ADD CONSTRAINT fk_supplier_comp + FOREIGN KEY (supplier_id,supplier_name) + REFERENCES supplier(supplier_id,supplier_name)... + - Fix import of BLOB data using INSERT statements into the bytea. + Thanks to rballer for the patch. + - Fix missing export of FK when no schema is provided. + +2016 01 13 - v16.2 + +This release fixes several issues, is more accurates on migration +assessment report and adds some new ora2pg command line options: + + * Add --pg_dsn, --pg_user and --pg_pwd to be able to set the + connection to import directly into PostgreSQL at command line. + * Add -f option to script import_all.sh to force to not check + user/database existing and skip their creation. + +Potential backward compatibility issues: + + * PG_SUPPORTS_CHECKOPTION is now enabled by default, you may want + to migrate to PostgreSQL 9.4 or above. + * Remove modification of CLIENT_ENCODING in generic configuration + file with --init_project, use the default instead. + * Remove modification of directive NLS_LANG to AMERICAN_AMERICA.UTF8 + in generic configuration file with --init_project, use the default + instead. + +Here is the complete list of other changes: + + - Adjust DBMS_OUTPUT calls to the migration assessment count. + - Fix migration assessment count of call to cursor%ISOPEN and + cursor%ROWCOUNT. + - Replace zero date also with prepared statement with online + PostgreSQL import and INSERT action. Thanks to Sebastian Albert + for the report. + - Remove REFERENCING clause in conditional triggers. Thanks to Raqua + for the report. + - Fix position of TG_OP condition when an exception is defined. + Thanks to Raqua for the report. + - Fix wrong replacement of SELECT with PERFORM when a comment was + found between an open parenthesis and the select statement. + Thanks to Raqua for the report. + - Fix procedure return type with OUT and INOUT parameter. Thanks to + Raqua for the report. + - Fix rewrite of triggers with referencing clause. Thanks to Raqua + for the report. + - Fix default number of --human_days_limit in usage. + - Fix replacement of placeholder %TEXTVALUE-d% to hide text string + in query during function call rewrite. Thanks to Lorena Figueredo + for the report. + - Fix progress bar when a WHERE clause is used to limit the number + of row to export. + - Fix error "DBD::Pg::db do failed: SSL error: decryption failed or + bad record mac" with pararellel table export (-P) and direct + import to PostgreSQL via a ssl connection. Thanks to pbe-axelor + for the report. + - Fix missing index name in indexes creation. Thanks to Raqua for + the report. + - Fix pg DSN in import_all.sh autogenerated script. + - Fix extraction of trigger. When the name of a column or something + contained INSERTING, DELETING or UPDATING was converted to TG_OP + = 'INSERT' or corresponding event. Thanks to Stanislaw Jankowski + for the patch. + - Fix multiple use of same column in check constraint and indexes + of partitions when there was several schema with the same objects. + - Fix default value for HUMAN_DAY_LIMIT to 5 when it is not defined + in ora2pg.conf. + - Fix double quote on column name in COPY export of partition tables + Thanks to Chris Brick for the report. + - Prevent case with several time same column in multicolumns unique + constraints. Fix typo in previous patch. + - Fix double quoted name with auto incremented sequence exported as + serial. + - Fix syntax error with MySQL data export with a WHERE clause using + LIMIT. + +2015 11 30 - v16.1 + +This release fixes several issues and adds some very useful features: + + * Generate automatically a new import_all.sh shell script when using option + --init_project to help automate all import into PostgreSQL. + + See sh import_all.sh -? for more information. + + * Export Oracle bitmap index as PostgreSQL btree_gin index. This require the + btree_gin extension and PostgreSQL >= 9.4. This is the default. + + * Auto set DEFINED_PK to the first column of a table that have a unique key + defined that is a NUMBER. This allow data of any table with a numeric + unique key to be extracted using multiple connexions to Oracle using -J + option. Tables with no numeric unique key will be exported with a single + process. + + * Improve BLOB export speed by using hex encoding instead of escape. This + might speed up be BLOB export by 10. + + * Allow use of LOB locator to retrieve BLOB and CLOB data to prevent having + to set LONGREADLEN. Now LONGREADLEN is set to 8KB. Old behavior using + LONGREADLEN can still be enabled by setting NO_LOB_LOCATOR to 0, given + for backward compatibility. Default is to use LOB locator. + + * Ora2Pg will also auto detect table with BLOB and automatically decrease + DATA_LIMIT to a value lower or equal to 1000. This is to prevent OOM. + + * Improving indexes and constraints creation speed by using the LOAD action + and a file containing SQL orders to perform. It is possible to dispatch + those orders over multiple PostgreSQL connections. To be able to use this + feature, PG_DSN, PG_USER and PG_PWD must be set. Then: + + ora2pg -t LOAD -c config/ora2pg.conf -i schema/tables/INDEXES_table.sql -j 4 + + will dispatch indexes creation over 4 simultaneous PostgreSQL connections. + + This will considerably accelerate this part of the migration process with + huge data size. + + * Domain indexes are now exported as b-tree but commented to let you know + where possible FTS are required. + + * Add number of refresh ON COMMIT materialized view in detailed report. + + * Allow redefinition of numeric type, ex: NUMBER(3)::bigint to fix wrong + original definition in Oracle. + + * Allow export of all schemas from an Oracle Instance when SCHEMA directive + is empty and EXPORT_SCHEMA is enabled. All exported objects will be + prefixed with the name of their original Oracle schema or search_path will + be set to that schema name. Thanks to Magnus Hagander for the feature + request. + + * Allow use of COPY FREEZE to export data when COPY_FREEZE is enabled. This + will only works with export to file and when -J or ORACLE_COPIES is not + set or default to 1. It can be used with direct import into PostgreSQL + under the same condition but -j or JOBS must be unset or default to 1. + Thanks to Magnus Hagander for the feature request. + +Some new configuration directives: + + * BITMAP_AS_GIN: enable it to use btree_gin extension to create bitmap + like index with pg >= 9.4. You will need to create the extension by + yourself: "create extension btree_gin;". Default is to create GIN index, + when disabled, a btree index will be created. + * NO_LOB_LOCATOR: to disable use of LOB locator and extract BLOB "inline" + using a less or more high value in LONGREADLEN. + * BLOB_LIMIT: to force the value of DATA_LIMIT for tables with BLOB. Default + is to automatically set this limit using the following code: + BLOB_LIMIT=DATA_LIMIT; while (BLOB_LIMIT > 1000) BLOB_LIMIT /= 10 + * COPY_FREEZE: use it to use COPY FREEZE instead of simple COPY to speedup + import into PostgreSQL. + +Here is the complete list of other changes: + + - Limite package function name rewrite to call with parenthesis after the + function name to avoid rewriting names of other objects. + - Fix extra replacement of function name with package prefix. On some + condition it was done multiple time. + - Set REPLACE_ZERO_DATE to -INFINITY in generic configuration when --mysql + is enabled. + - Fix extraction of partition with MySQL that was not limited to a single + database. + - Do some replacement on ORACLE_DNS and SCHEMA into generic configuration + when --mysql is used for better understanding. + - Add call to round() on -J parallelization when the auto detected column + is a numeric with scale. + - Add COMMIT to the difficulties migration assessment keywords as it need + context analyzing. + - Add call to cursor's %ROWCOUNT, %ISOPEN and %NOTFOUND to difficulties + migration assessment keywords. + - Replace call to CURSOR%ROWTYPE by RECORD. Thanks to Marc Cousin for the + report. + - Fix ALTER FUNCTION ... OWNER TO ... and REVOKE statement where functions + parameters were missing. + - Add Get_Env to the Oracle functions list for migration assessment. + - Disable variable NO_LOB_LOCATOR and set LONGREADLEN to 8192 bytes to use + LOB locators to extract BLOB in generic configuration file. + - Fix call method "disconnect" on unblessed reference at line 9998. Thanks + to Stephane Tachoires for the report. + - Exclude from export objects name matching /.*\$JAVA\$.*/ and /^PROF\$.*/. + - Fix migration assessment report when created during the package export. + - Force writing Oracle package body in separate files when FILE_PER_FUNCTION + is enabled and PLSQL_PGSQL disable to obtain package source code. + - Fix case where sequence max value is lower than start value, in this case, + set max value = start value. + - Fix missing newline after each package file to import in global package.sql + file when FILE_PER_FUNCTION is enabled. + - Remove export of user PUBLIC in GRANT export. + - Set DISABLE_TRIGGERS to 1 in generic configuration file auto generated when + ora2pg option --init_project is used. + - Remove call to quote_reserved_words() with index column when it we detect + a function based index, too much false positive are rewritten with SQL code + like CASE...WHEN. + - Update export_schema.sh to remove .sql files when there is not such object + leaving export directory empty. + - Prevent creating TBSP_INDEXES_tablespace.sql when no tablespaces are found + - Update documentation on WHERE clause on how to limit the number of tuples + exported for Oracle and MySQL to test data import. + - Fix unlisted spatial indexes in assessment report. + - Fix double quote on index name with index renaming and reserved keyword. + - Do not try to export tablespaces, privileges and audited queries as non DBA + user when USER_GRANT is enabled. + - Remove carriage return from list file. + - Force SCHEMA to database name with MySQL migration. + - Fix missing declaration of _extract_sequence_info(). Thank to Yannick DEVOS + for the report. + - Add documentation about COPY_FREEZE directive and add a note about export + of all schema. + - Remove systematic schema name appended to table name on KETTLE export, this + must only be true when EXPORT_SCHEMA is enabled. + - Fix TO_NUMBER() wrong replacement when a function is called as a parameter. + - Fix non converted DECODE() when they was called in an XMLELEMENT function. + - Suppress MDSYS.SDO_* from MDSYS call in migration assessment cost. + - Remove use of DBMS_STANDARD called with raise_application_error function + - Fix STRING type replacement + - Recreate README as a text file, not a man page. + - Reformat changelog to 80 characters. + - Add -t | --test command line option to ora2pg_scanner to be able to test + all connections defined into the CVS list file. + +2015 10 15 - v16.0 + +This major release improve PL/SQL code replacement, fixes several bugs and +adds some major new features: + + * Full migration of MySQL database, it just work like with Oracle database. + * Full migration assessment report for MySQL database. + * New script, ora2pg_scanner, to perform a migration assessment of all + Oracle and MySQL instances on a network. + * Add technical difficulty level in migration assessment. + * Allow migration assessment on client queries extracted from AUDIT_TRAIL + (oracle) or general_log table (mysql). + * Ora2Pg has a "made in one night" brand new Web site (still need some work). + See http://ora2pg.darold.net/ + +Example of technical difficulty level assessment output for the sakila database +with some more difficulties: + + Total 83.90 cost migration units means approximatively 1 man-day(s). + Migration level: B-5 + +Here are the explanation of the migration level code: + + Migration levels: + A - Migration that might be run automatically + B - Migration with code rewrite and a human-days cost up to 10 days + C - Migration with code rewrite and a human-days cost above 10 days + Technical levels: + 1 = trivial: no stored functions and no triggers + 2 = easy: no stored functions but with triggers, no manual rewriting + 3 = simple: stored functions and/or triggers, no manual rewriting + 4 = manual: no stored functions but with triggers or views with code + rewriting + 5 = difficult: stored functions and/or triggers with code rewriting + +This is to help you to find the database that can be migrated first with small +efforts (A and B) and those who need to conduct a full migration project (C). + +This release has also some new useful features: + + * Export type SHOW_TABLE now shows additional information about table type + (FOREIGN, EXTERNAL or PARTITIONED with the number of partition). + * Connection's user and password can be passed through environment variables + ORA2PG_USER and ORA2PG_PASSWD to avoid setting them at ora2pg command line. + * Improve PL/SQL replacement on ADD_MONTH(), ADD_YEAR(), TRUNC(), INSTR() and + remove the replacement limitation on DECODE(). + * Add detection of migration difficulties in views, was previously reserved + to functions, procedures, packages and triggers. + * Replace values in auto generated configuration file from command line + options -s, -n, -u and -p when --init_project is used. + * Adjust lot of scores following new functionalities in Ora2Pg, ex: dblink or + synomyms are now easy to migrate. + +There is some new command line options to ora2pg script: + + * -m | --mysql : to be used with --init_project and -i option to inform + ora2pg that we work with a MySQL format + * -T | --temp_dir : option to be able to set a distinct temporary directory + to run ora2pg in parallel. + * --audit_user : option to set the user used in audit filter and enable + migration assessment report on queries from AUDIT_TRAIL (oracle) or + general_log table (mysql). + * --dump_as_sheet and --print_header options to be able to compute a CSV + file with all migration assessment from a list of oracle database. + * --dump_as_csv option to report assessments into a csv file. It will not + include comments or details, just objects names, numbers and cost. + +Backward compatibility: + + - Change NULL_EQUAL_EMPTY to be disabled by default to force change in the + application instead of transforming the PL/SQL. + +This release adds some new configuration directives: + + * MYSQL_PIPES_AS_CONCAT: Enable it if double pipe and double ampersand + (|| and &&) should not be taken as equivalent to OR and AND. + * MYSQL_INTERNAL_EXTRACT_FORMAT: Enable it if you want EXTRACT() replacement + to use the internal format returned as an integer. + * AUDIT_USER: Set the comma separated list of user name that must be used + to filter from the DBA_AUDIT_TRAIL or general_log tables. + * REPLACE_ZERO_DATE: "zero" date: 0000-00-00 00:00:00 it is replaced by a + NULL by default, use it to use the date of your choice. + * INDEXES_RENAMING: force renaming of all indexes using tablename_columnsname + Very useful for database that have multiple time the same index name or + that use the same name than a table. + * HUMAN_DAYS_LIMIT: default to 5 days, used to set the number of human-days + limit for migration of type C. + +Here is the full list of other changes: + + - Remove list of acknowledgment that was not maintained anymore and some + person may feel injured. Acknowledgment for patches or bug reports are + always written to changelog, so this part reports to it now. + - Fix bad trigger export when objects was enclosed in double quote and fix + an additional bug in WHEN clause export. Thanks to Cyrille for the report. + - Update documentation. + - Update Makefile.PL with new script to install and new configuration + directives in auto generated configuration file. + - Update with new and missing files. + - Add a Perl Module dedicated to MySQL database object discovery and export, + lib/Ora2Pg/MySQL.pm. + - Fix function based index type replacement in previous commit. + - Do not report indexes with just DESC as function based index like Oracle + report it. Thanks to Marc Cousin for the report. + - Some excluded table was missing in the previous patch. + - Remove use of DBI InactiveDestroy call when a fork is done and replace it + to a single use AutoInactiveDestroy at connection. This require DBI>=1.614. + - Add SDO_* tables and OLS_DIR_BUSINESSES in table exclusion list to fix issue + #124 when no schema is provided. Thanks to Kenny Joseph for the report. + - Fix partition prefix. + - Remove UNIQUE keyword from spatial index. + - Fix alter triggers function with missing parenthesis. Thanks to Spike Hodge + MWEB for the report. + - Fix export of foreign keys when they was defined in lowercase. Thanks to + Spike for the report. + - Fix wrong offset when rewriting ROWNUM with LIMIT+OFFSET. Thanks to Marc + Cousin for the report. + - Allow -INFINITY to be used to replace zero date. + - Migration assessment in hour-day are now set to 1 man-day, we do not need + such a precision and it is easier to process csv report. Thanks to Stephane + Tachoire for the report. + - Fix some issue with FDW and WKT spatial export. Add migration assessment + of queries from the AUDIT_TRAIL table. + - Adjust assessment units of some objects and add QUERY migration weight. + - Rewrite information about migration levels. + - Fix speedometer in progress bar, it will now shows the current speed in + tuples/sec and the speed and time related to a table when export ended for + the object. Thanks to Alex Ignatov for the report. + - Fix break line when export data using INSERT mode. Thanks to Vu Bui for + the report. + - Do not display line about non existent objects in migration assessment + reports. + - Fix date default value for date when value is 0000-00-00 00:00:00 + - Suppress display of title for function and trigger details when there is + no details. + - Remove INSTR() from the list of Oracle function that are not supported. + It is now replaced by position(). + - Fix condition to call _get_largest_tables(). + - Fix some minor issues in OUT/INOUT type returned by a function. + - Fix default value that may appears unquoted. + - Fix several issues on partition export: column with function, index on + default partition table and plsql to plpgsql translation in check condition. + - Fix some minor issues. + - Replace values from command line options -s, -n, -u and -p in --init_project + auto generated configuration file. Thanks to Stephane Tachoire for the + feature request. + - Fix wrong object count in SHOW_REPORT. Thanks to Stephane Tachoire for + the report. + - Use DBA_SEGMENTS to find database size when USER_GRANT is disable, aka user + is a DBA + - Remove report of Migration Level when --estimate_cost is not enabled. + - Add missing BINARY_INTEGER for type replacement. + - Always exclude function squirrel_get_error_offset() that is created by the + Squirrel Oracle plug-in. + - Adjust assessment scores following new functionalities in Ora2Pg, ex: + autonomous transaction, dblink or synomyms are now easy to migrate. + - Remove man page from source, it is auto generated by Makefile.PL and make. + - Fix unterminated DECODE replacement when there was more than 5 parameters + to DECODE() and remove the limitation to 10 parameters. There is no more + limit in the number of decode parameters. Thanks to Mael Rimbault for the + report. + - Remove inclusion of unwanted object when exporting a limited list of view + with ALLOW. + - Disable unsupported recursive query used to reorder views when Oracle + version is 11gR1. Thanks to Mael Rimbault for the patch. + - Add PLPGSQL replacement of INSTR() by POSITION(). Thanks to Mael Rimbault + for the report. + - Add difficulty level information in migration assessment, this include a + new configuration directive HUMAN_DAYS_LIMIT (default to 5 days) to set + the number of human-days limit for migration of type C. + - Add MERGE with a migration cost of 3, still need work be replaced by + ON CONFLICT. + - Remove some redundant regular expressions. + - Fix escaped commas not working properly in MODIFY_TYPE. A MODIFY_TYPE + value like `TABLE1:COL4:decimal(9\,6)` was leading to a column like + `col4 decimal(8#nosep#3)` in the SQL dump file that was generated. This + fixes the output to be `col4 decimal(8,3)`. Thanks to Nick Muerdter for + the patch. + - Strip default "empty_clob()" values from table dumps. This function does + not exist in Postgres and is not necessary. Thanks to Nick Muerdter for + the patch. + - Fix undesired double quoting of column name in function based indexes. + - Fix issue with Perl < 5.8 "Modification of a read-only value attempted" + - Fix retrieving of table size on Oracle 8i. + - Add auto double quoting of object name with unauthorized characters. + Thanks to Magnus Hagander for the feature request. + - Automatically double quote object name beginning with a number + - Fix missing DESC part in descending indexes. Thanks to Magnus Hagander + for the report. + - Fix case where a column name in oracle is just a number (e.g. the column + is called "7"), it will be created in postgres without quoted identifier, + which fails. Thanks to Magnus Hagander for the report. + - Fix "reqs/sec" display in debug mode. Thanks to Laurent Martelli for + the patch + - Fix export if Oracle procedure is created without a parameter. Thanks to + dirkgently007 for the report. + - Fix CSV report output. + - Fix triggers from file parser. + - Add a test on triggers return to handle case where it is triggered on + DELETE + other(s) event(s). In this case a test is done on the TG_OP to + return OLD if event is DELETE or NEW in other case. Thanks to Dominique + Legendre for the suggestion. + - Change NULL_EQUAL_EMPTY to be disabled by default to force change of the + application instead of transforming the PL/SQL. + - Change score of SYNONYM and DBLINK in the migration assessment. + - Add conversion of Oracle type STRING into varchar(n) or text. + - Add information about libaio1 requirement for instant client + - Remove extra space when calling ora2pg_get_efile() used to export BFILE + into EFILE. Thanks to Dominique Legendre for the export. + + +2015 06 01 - v15.3 + +This is a maintenance release only that fixes several minor bugs and typos. +The configuration file have been entirely rewritten to classify configuration +directives in section for better understanding. + +Here is the full list of changes: + + - Ora2Pg will use DEFAULT_SRID when call to sdo_cs.map_oracle_srid_to_epsg() + returns an ORA-01741 error. Mostly because there's no SRID defined for that + column in ALL_SDO_GEOM_METADATA. The error message will still be displayed + but a warning will explain the reason and ora2pg will continue with default + value. Thanks to kazam for the report. + - Add current setting for NLS_TIMESTAMP_FORMAT and NLS_DATE_FORMAT to the + SHOW_ENCODING report. + - Change default value for GEOMETRY_EXTRACT_TYPE to INTERNAL instead of WKT. + - Change generic configuration file behavior with BINMODE parameter commented + if it was previously uncommented. This will force to use the default value. + - Fix potential issue with max open file limit with unclosed temporary file. + Thanks to Marc Clement for the report. + - Fix use of SECURITY DEFINER in SYNONYM export. + - Fix parsing of editable function/procedure/package from input DML file. + - Fix case where variable $2 and $3 was null after a too early call of a new + substitution in read_view_from_file(). Thanks to Alex Ignatov for the patch. + - Add support to "create or replace editionable|noneditionable" from input DML + files. Thanks to Alex Ignatov for the report. + - Fix unknown column HIGH_VALUE from *_TAB_PARTITIONS in Oracle 8i. Thanks to + Sebastian Fischer for the patch. + - Fix call to ALL_MVIEW_LOGS object which not exists with Oracle 8i. Thanks to + Sebastian Fischer for the report. + - Fix Error ORA-22905 while Get the dimension of the geometry by looking at + number of element in the SDO_DIM_ARRAY. Thanks to jkutterer for the patch. + - Remove reordering export of view for Oracle database before 11g. Thanks to + kyiannis for the report. + - Fix several some typos and a bunch of misspelled. Thanks to Euler Taveira + for all the patches. + - Fix missing Oracle database version before looking at function security + definer. Thanks to kyiannis for the report. + +2015 04 13 - v15.2 + +This new minor release fixes some issues and adds two new configuration +directives: + + * ORA_INITIAL_COMMAND to be able to execute a custom command just after + the connection to Oracle, for example to unlock a security policy. + * INTERNAL_DATE_MAX to change the behavior of Ora2Pg with internal date + found in user defined types. + +This version will also automatically re-order exported views taking into +account interdependencies. + +Here is the full list of changes: + + - Add INTERNAL_DATE_MAX configuration directive with default to 49 to be + used when reformatting internal date returned with a user defined type + and a timestamp column. DBD::Oracle only return the internal date format + 01-JAN-77 12.00.00.000000 AM so it is difficult to know if the year value + must be added to 2000 or 1900. We takes the default behavior where date + are between 1950 and 2049. + - Remove extra CHAR and BYTE information from column type. Thanks to Magnus + Hagander for the report. + - Re-order views taking into account interdependencies. Thanks to Kuppusamy + Ravindran and Ulrike for the suggestion and the Oracle query. + - Fix case sensitivity in function based indexes. Thanks to Kuppusamy + Ravindran for the report. + - Fix PERFORM wrong replacement and infinite loop processing DECODE in some + condition. Thanks to Didier Brugat for the report. + - Fix replacement of boolean value in DEFAULT value at table creation. + Thanks to baul87 for the report. + - Add ORA_INITIAL_COMMAND configuration directive to be able to execute a + custom command just after the connection to Oracle, to unlock a policy for + example. Thanks to Didier BRUGAT for the feature request. + - Fix alias in from clause when an XML type is found. Thanks to Lance Jacob + for the record. + - Invert condition on excluding temporary file with Windows OS. Thanks to + kazam for the report. + - Remove start time and global number of rows from _dump_table() parameters + they are not used anymore. + - Remove use of temporary file on Windows operating system. + - Disable parallel table export when operating system is Windows. + - Fix export of objects with case sensitivity using ALLOW or EXCLUDE + directives. Thanks to Alexey Ignatov for the report. + - Fix export of triggers from recycle bin. + - Fix count of synonym in assessment report. + - Add list of tables created by OEM to the exclusion list. + - Fix look at default configuration file and set mode of export_schema.sh + to executable by default. Thanks to Kuppusamy Ravindran for the report. + - Add AUTHORIZATION to the list of PostgreSQL reserved word. Thanks to + Kuppusamy Ravindran for the report. + - Display a warning when an index has the same name as the table itself so + that you can renamed it before export. Thanks to Kuppusamy Ravindran for + the feature request. + - Fix export of function based indexes with multiple column. Thanks to + Kuppusamy Ravindran for the report. + - Modify ora2pg script to return 0 on success, 1 on any fatal error and 2 + when a child process die is detected. + - Change the way the generic configuration file is handle during project + initialization. You can use -c option to copy your own into the project + directory. If the file has the .dist extension, ora2pg will apply the + generic configuration on it. Thanks to Kuppusamy Ravindran for the report + and features request. + - Add debug information when cloning the Oracle connection. + - Force return of OLD when the trigger is on DELETE event + +2015 02 06 - v15.1 + +New minor release just to fix two annoying bugs in previous release. + + - Fix replacement of function name which include SELECT in their name by + PERFORM. Thanks to Frederic Bamiere for the report. + - Fix creation of sources subdirectories when initializing a new migration project. + +2015 02 04 - v15.0 + +This major release improve PL/SQL code replacement, fixes several bugs and +adds some new useful features: + + - Add support to the PostgreSQL external_file extension to mimic BFILE + type from Oracle. See https://github.com/darold/external_file for + more information. + - Allow export of Oracle's DIRECTORY as external_file extension objects + This will also try to export read/write privilege on those directories. + - Allow export of Oracle's DATABASE LINK as Oracle foreign data wrapper + server using oracle_fdw. + - Allow function with PRAGMA AUTONOMOUS_TRANSACTION to be exported through + a dblink wrapper to achieve the autonomous transaction. + - Allow export of Oracle's SYNONYMS as views. Views can use foreign table + to create "synonym" on object of a remote database. + - Add trimming of data when DATA_TYPE is used to convert CHAR(n) Oracle + column into varchar(n) or text. Default is to trim both side any space + character. This behavior can be controlled using two new configuration + directives TRIM_TYPE and TRIM_CHAR. + - Add auto detection of geometry constraint type and dimensions through + spatial index parameters. This avoid the overhead of sequential scan + of the geometric column. + - Add support to export Oracle sub partition and create sub partition + for PostgreSQL with the corresponding trigger. + - ALLOW and EXCLUDE directives are now able to apply filter on the object + type. Backward compatibility can not be fully preserved, older definition + will apply to current export type only, this could change your export in + some conditions. See documentation update for more explanation. + - Add PACKAGE_AS_SCHEMA directive to change default behavior that use a + schema to emulate Oracle package function call. When disable, all calls + to package_name.function_name() will be turn into package_name_function_name() + just like a function call in current schema. + - Add FKEY_OPTIONS to force foreign keys options. List of supported options + are: ON DELETE|UPDATE CASCADE|RESTRICT|NO ACTION. + - Add rewriting of internal functions in package body, those functions will + be prefixed by the package name. Thanks to Dominique Legendre for the + feature request. + +Some change can break backward compatibility and make configuration directives +obsolete: + + - The ALLOW_PARTITION configuration directive has been removed. With new + extended filters in ALLOW/EXCLUDE directive, this one is obsolete. + Backward compatibility is preserved but may be removed in the future. + - ALLOW and EXCLUDE directives do not works as previously. Backward + compatibility may be preserved with some export type but may be broken + in most of them. See documentation. + - It is recommended now to leave the NLS_LANG and CLIENT_ENCODING commented + to let Ora2Pg handle automatically the encoding. Those directives may be + removed in the future. + +Here is the full changelog of the release: + + - Declares SYNONYM views as SECURITY DEFINER to be able to grant access to + objects in other schema. + - Fix wrong replacement of data type in function body. Thanks to Dominique + Legendre for the report. + - Fix missing column name replacement on trigger export when REPLACE_COLS + is defined. Thanks to Dominique Legendre for the report. + - Fix missing table replacement on trigger export when REPLACE_TABLES is + defined. Thanks to Dominique Legendre for the report. + - Fix case where IS NULL substitution was not working. Thanks to Dominique + Legendre for the report. + - Remove double exclusion clause when multiple export type is used with same + column name and no values defined. + - Allow parsing of DATABASE LINK and SYNONYM from a DDL file. + - Add DIRECTORY export type to export all Oracle directories as entries for + the external_file extension. This will also export read/write privilege + on those directories. Thanks to Dominique Legendre for the feature request. + - Review documentation about NULL_EQUAL_EMPTY. + - Fix missing code to replace IS NULL as coalesce(...). Thanks to Dominique + Legendre for the report. + - Add external_file schema to search_path when BFILE is set to EFILE in + directive DATA_TYPE. Thanks to Dominique Legendre for the request. + - Remove IF EXIST clause to oracle function created by Ora2Pg for BFILE + export. Thanks to Dominique Legendre for the report. + - Add support to the PostgreSQL external_file extension to mimic BFILE type + from Oracle. See https://github.com/darold/external_file for more information. + - Add auto detection of geometry constraint type and dimensions through the + spatial index parameters first. This avoid the overhead of sequential scan + of the geometric column. + - Remove lookup at package function when not required. + - Fix issue with database < 10g that do not have the DROPPED column into the + ALL_TABLES view. Thanks to Lance Jacob for the report. + - Add trimming of data when DATA_TYPE is used to convert CHAR(n) Oracle + column into varchar(n) or text column into PostgreSQL. Default is to + trim both side any whitespace character. This behavior can be controlled + using the new configuration directives TRIM_TYPE and TRIM_CHAR. + - Update copyright year. + - Add assessment cost for object TABLE SUBPARTITION and review cost for + object DATABASE LINK. + - Update documentation about SYNONYM export. + - Allow export of SYNONYMS as views with a new export type: SYNONYM. + - Fix object exclusion function with Oracle 8i and 9i. Thanks to Lance Jacob + for the report. + - Fix INTERVAL YEAR TO MONTH and DAY TO SECOND with precision. + - Remove unused pragma from the cost assessment. + - Suppress PRAGMA RESTRICT_REFERENCES, PRAGMA SERIALLY_REUSABLE and INLINE + from the PLSQL code. There is no equivalent and no use in plpgsql. + - Fix several issues in function/procedure/package extraction from file + input and some other related bug. + - Remove single slash and \\r from function code. + - Remove schema from package name with input file to avoid creating + function with SCHEMA.PKGNAME.FCTNAME + - Fix ALLOW/EXCLUDE ignored with type COPY or INSERT. Thanks to thleblond + for the patch. + - Fix setting of NLS_NUMERIC_CHARACTERS and NLS_TIMESTAMP_FORMAT with + multiprocess, the session parameters was lost with the cloning of the + database handle. Thanks to thleblond for the patch. + - Fix issue that could produce errors "invalid byte sequence" when dumping + data to pg database by forcing the client_encoding when PG_DSN is set. + Thanks to thleblond for the patch. + - Fix issue to add parenthesis with function with no parameters and wrong + use of PERFORM in cursor declaration. Thanks to hdeadman for the report. + - Fix broken export of function or procedure without parameter in package + body. Thanks to hdeadman for the report. + - Fix ERROR: "stack depth limit exceeded" generated by an infinite loop in + partition trigger when there is no default table when value is out of range. + - Add support to Oracle sub partition export. + - Fix issue with procedure in package without parameters. + - Enable DISABLE_SEQUENCE in generic configuration file. + - Fix unwanted alter sequence in data export when there is table allowed + or excluded. + - Fix initial default values of command line parameter that prevent value + in configuration file to be taken. + - Fix non working global definition of table in ALLOW and EXCLUDE directive + with COPY and INSERT export. + - Update ora2pg.spec, thanks to bbuechler for the patch. + - Close temporary files before deleting them, on Windows if they are not + explicitly closed there are not deleted. Thanks to spritchard for the + patch. + - Force schema name to be uppercase when PRESERVE_CASE is disable (default). + Thanks to Jim Longwill for the report. + - Add rewriting of internal functions in package body, those functions will + be prefixed by the package name. Thanks to Dominique Legendre for the + feature request. + - Fix type replacement in user defined type. Thanks to Dominique Legendre + for the report. + - Add filter with INSTEAD OF triggers on views to TRIGGER export type. Thanks + to Dominique Legendre for the feature request. + - Fix replacement of function name when PACKAGE_AS_SCHEMA is disabled. + - Fix PLSQL_PGSQL that was always set to 0 when -p was not used even if + configuration directive PLSQL_PGSQL was activated. Thanks to Dominique + Legendre for the report. + - Remove ALTER SCHEMA ... OWNER TO ... when CREATE_SCHEMA is not enable. + Thanks to Dominique Legendre for the report. + - Add DBLINK export to be created as foreign data wrapper server. Thanks to + the BRGM for the feature request. + - Remove ALLOW_PARTITION configuration directive, with extended filter in + ALLOW/EXCLUDE directive, this one is obsolete. Backward compatibility is + preserved. + - Add documentation about extended filters in ALLOW and EXCLUDE directive. + - Update documentation about VIEW_AS_TABLE and remove statement change with + export TYPE is VIEW. + - Add filter to grant export on functions, sequences, views, etc. + - Fix GRANT in ALLOW or EXCLUDE filters. + - Add commented order: "REVOKE ALL ON FUNCTION ... FROM PUBLIC;" when the + function is declared as SECURITY DEFINER. + - Prevent collecting column information with SHOW_TABLE export type. + - Fix default value SYSTIMESTAMP to CURRENT_TIMESTAMP, and remove DEFAULT + empty_blob(). Thanks to hdeadman for the report. + - ALLOW and EXCLUDE directives are now able to apply filter on the object + type. Backward compatibility can not be fully preserved, older definition + will apply to current export type only, this could change your export in + some conditions. See documentation update for more explanation. Thanks to + the BRGM for the feature request. + - Force function to be created with SECURITY DEFINER when AUTHID in table + ALL_PROCEDURES is set to DEFINER in Oracle. This only works with Oracle + >= 10g. Thanks to Dominique Legendre for the feature request. + - Add PACKAGE_AS_SCHEMA configuration directive to change default behavior + to use a schema to emulate Oracle package function call. When disable all + call to package_name.function_name() will be turn into package_name_function_name() + just like a function call in current schema. Thanks to the BRGM for the + feature request. + - Add a note to documentation about the way to convert srid into Oracle + database instead of in Ora2Pg. Thanks to Dominique Legendre for the hint. + - Fix documentation about SHOW_ENCODING export type. + - Remove use of REGEX_LIKE with Oracle version 9. Thanks to Lance Jacob for + the report. + - Replace new FKEY_OPTIONS by FKEY_ADD_UPDATE configuration directive with + three possible values: always, never and delete. It will force or not + Ora2Pg to add "ON UPDATE CASCADE" on foreign keys declaration. + - Allow FORCE_OWNER to work with all exported objects. Thanks to BRGM for + the feature request. + - Add FKEY_OPTIONS to force foreign keys options. List of supported options + are: ON DELETE|UPDATE CASCADE|RESTRICT|NO ACTION. Thanks to the BRGM for + the feature request. + - Fix ambiguous column in view extraction. Thanks to Dominique Legendre for + the report. + - Fix replacement of TYPE:LEN by boolean, ex: REPLACE_AS_BOOLEAN CHAR:1. + Thanks to jwiechmann for the report. + - Fix error ORA-00942 where Ora2Pg try to export data from a view defined + in VIEW_AS_TABLE configuration directive. + - Update list of excluded Oracle schema to the documentation. + - Fix export of all views with comments when VIEW_AS_TABLE is set. + - Fixed some typos in the generated sample configuration file. Thanks to + Hal Deadman for the patch. + - Limit column information export to the type of object extracted. + - Remove call to MDSYS in SQL code. Thanks to Dominique Legendre for the + report. + - Add more Oracle schema to the exclusion list. + - Fully remove join on DBA_SEGMENTS to retrieve the list of tables, views + and comments. Replaced by ALL_OBJECTS. Thanks to Dominique Legendre for + the help. + - Exclude JAVA\$.* tables and fix tables list query to include newly created + tables with no segments. Thanks to Dominique Legendre for the fix. + - Fix regex that convert all x = NULL clauses to x IS NULL to not replace + := NULL too. + - Autodetect unusual characters in owner name when extracting data and used + it embeded into double quote. + - Replace single return with return new in trigger code. Thanks to Dominique + Legendre for the report. + +2014 11 12 - v14.1 + +This is a maintenance release only mainly to add patches that was not +been applied in previous major release. + + - Remove ALLOW_CODE_BREAK, it is no more useful. + - Change output of SHOW_ENCODING to reflect change to default encoding. + - Comment ALLOW_PARTITION in default configuration file + - Add QUERY and KETTLE export type in configuration file comments. + +2014 11 05 - v14.0 + +This major release adds full export of Oracle Locator or Spatial geometries into +PostGis, SDO_GEOM functions and SDO_OPERATOR are also translated. This export +adds the following features: + + 1. Basic and complex geometry types support + 2. Geometry data conversion from Oracle to PostGIS + 3. Spatial Index conversion + 4. Geometry metadata / constraints support + 5. Spatial functions conversion + +For spatial data export, you have three choice, WKT to export data using +SDO_UTIL.TO_WKTGEOMETRY(), WKB to export data using SDO_UTIL.TO_WKBGEOMETRY() +and INTERNAL to export geometry using a Pure Perl library. Unlike the first +two methods, INTERNAL is fast and do not raise Out Of Memory. The export is +done in WKT format so that you can verify your geometry before importing to +PostgreSQL. + +Other additional major features are: + + - Parallel table processing. + - Auto generation of migration template with a complete project tree. + - Allow user defined queries to extract data from Oracle. + +Parallel table processing is controlled by the -P or --parallel command line +options or the PARALLEL_TABLE configuration directive to set the number of +tables that will be processed in parallel for data extraction. The limit is +the number of cores on your machine. Ora2Pg will the open one connection to +Oracle database for each parallel table extraction. This directive, when upper +than 1, will invalidate ORACLE_COPIES but not JOBS, so the real number of +process that will be used is (PARALLEL_TABLES * JOBS). + +The two options --project_base and --init_project when used indicate to Ora2Pg +to create a project template with a work tree, a generic configuration file +and a shell script to export all objects from the Oracle database. So that you +just have to define the Oracle database connection into the configuration file +and then execute the shell script called export_schema.sh to export your +Oracle database into files. Here a sample of the command and the project's tree. + + ora2pg --project_base /tmp --init_project test_project + + /tmp/test_project/ + config/ + ora2pg.conf + data/ + export_schema.sh + reports/ + schema/ + fdws/ functions/ grants/ kettles/ + mviews/ packages/ partitions/ + procedures/ sequences/ tables/ + tablespaces/ triggers/ types/ views/ + sources/ + functions/ mviews/ packages/ + partitions/ procedures/ triggers/ + types/ views/ + +It create a generic config file where you just have to define the Oracle +database connection and a shell script called export_schema.sh. The +sources/ directory will contains the Oracle code, the schema/ will +contains the code ported to PostgreSQL. The reports/ directory will +contains the html reports with the migration cost assessment. + +Sometime you may want to extract data from an Oracle table but you need a +custom query for that. Not just a "SELECT * FROM table" like Ora2Pg do but +a more complex query. The new directive REPLACE_QUERY allow you to overwrite +the query used by Ora2Pg to extract data. The format is TABLENAME[SQL_QUERY]. +If you have multiple table to extract by replacing the Ora2Pg query, you can +define multiple REPLACE_QUERY lines. For example: + + REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e + JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] + +Other new features are: + + - Export of declaration of language C function. Previous version was + not exporting function with no code body like external C function. + - Export of COMMENT from views. + - Function to replace some call to SYS_CONTECT(USERENV, ...) by the + PostgreSQL equivalent. + - Add POSTGIS_SCHEMA configuration directive to add the dedicated + PostGis schema into the search_path. + - Add PG_SUPPORTS_IFEXISTS configuration directive to be able to suppress + IF EXISTS call in DDL statement generated by Ora2Pg. + - Triggers are now all excluded/allowed following the table names specified + in the ALLOW and EXCLUDED directives + - Allow automatic export of nested tables (TYPE+TABLE+COPY). + +One change is not fully backward compatible: Ora2Pg now use UTF8 by default +on both side. On Oracle connection NLS_LANG is set to AMERICAN_AMERICA.AL32UTF8, +NLS_NCHAR to AL32UTF8. On PostgreSQL side CLIENT_ENCODING to UTF8. For export +that dump to files, Perl binmode is set to utf8. You can always change those +default setting in configuration file, but it is not recommanded. + +Here is the full changelog of the release: + + - Fix inline comments into function declaration. Thanks to Marcel Huber + for the report. + - Fix case where SELECT ... INTO was wrongly replaced by PERFORM. + - Fix DECODE() translation. Thanks to Dominique Legendre for the report. + - Add replacement of SDO_OPERATOR into PostGis relationships. + - Add replacement of SDO_GEOM spatial function to postgis ST_* functions. + - Add GEOMETRY_EXTRACT_TYPE configuration directive to specify the geometry + extracting mode: WKT (default), WKB and INTERNAL. + - Add a pure Perl library to export SDO_GEOMETRY as a WKT representation. + This is controlled by a new extraction type INTERNAL to use with the + GEOMETRY_EXTRACT_TYPE configuration directive. + - Remove USE_SC40_PACKAGE directive and any reference to this library, + it is not useful now that we have the INTERNAL geometry extraction mode. + - Fix replacement of varchar2 in PL/SQL function. + - Fix bug in type replacement when default values used function. + - Add export of declaration of language C function. Previous version was + not exporting function with no code body like external function. + - Fix create statement in export of view as table. Thanks to ntlis for the + report. + - Fix replacement of to_number without format. + - Add export of COMMENT from VIEWS. + - Add function to replace some call to SYS_CONTECT(USERENV, ...) by the + PostgreSQL equivalent. + - Fix parsing from file of tablespace. + - Fix wrong alias name in FROM clause when extracting XML data. Thanks + to Marc Sitges for the report. + - Fix export of comments in FDW export, might be COMMENT ON FOREIGN TABLE. + Thanks to David Fetter for the report. + - Fix broken export of function based indexes. Thanks to Floyd Brown for + the report. + - Fix sequence with negative minvalue/maxvalue and negative increment. + Thanks to jwiechmann for the report. + - Fix forced owner to schema to the value of FORCE_OWNER when it is set + to a user name. + - Fix create schema when FORCE_OWNER is enabled. Thanks to Dominique + Legendre for the report. + - Add POSTGIS_SCHEMA configuration directive to add a schema to the + search_path. Thanks to Dominique Legendre for the feature request. + - Returns NULL when a geometry is NULL instead of calling ST_AsText with + a null value. Thanks to Dominique Legendre for the report. + - Add more explanation about values of CONVERT_SID. + - Fix issue in DBMS_OUTPUT replacement. + - Fix exclusion of default objects from type export. + - When CONVERT_SRID is > 1 this value will be used to force the SRID value + on all export. + - Disable NULL_EQUAL_EMPTY in generic configuration when generating a project + tree. + - Add LOGMNR$ and RECAP$ in the exclusion objects list. + - Fix performance issue in extracting data from geometry column and add + AUDSYS,DVSYS and DVF to the list of schema to exclude. + - Prefix table name with schema name on queries for retrieving data to + avoid errors in multi schema export. + - Add SDO_* cost to migration report. + - Fix real number of Synonym that should be review. + - Fix wrong report of CTXSYS synonym. + - Enabled AUTODETECT_SPATIAL_TYPE by default. + - Remove KETTLE and FDW export from the auto generated project. + - Force the copy of /etc/ora2pg/ora2pg.conf.dist into the project directory + with no more look at the current ora2pg.conf. Force autodetection of + spatial type in the generic configuration. + - Huge performance gain on querying information about Spatial column. Thanks + to Dominique Legendre for the great help. + - Fix wrong use of table alias with SEGMENT_NAME. + - Add unified audit table (CLI_SWP$.*) from the exclusion list. + - Fix operator in check condition of range partitions. Thanks to Kaissa + Chellouche for the report. + - Add to the internal exclusion list tables generated by spatial indexes + MDRT_.*, sequences MDRS_.* and interMedia Text index DR$.*. Thanks to + Dominique Legendre for the report. + - Make REPLACE_TABLES and REPLACE_COLS work with VIEW. The view name and + the columns aliases will be replaced. Take care that the table name or + columns names in the statement will be kept untouched and need manual + rewriting. Thanks to Sven Medin for the feature request. + - Add PG_SUPPORTS_IFEXISTS configuration directive to be able to suppress + IF EXISTS call in DDL statement generated by Ora2Pg. PostgreSQL below + 9.x do not support this keywords. Thanks to George Kowalski fot the + feature request. + - Fix wrong substitution in EXECUTE ... USING statement, where parameters + number was not prefixed by a $ sign. Thanks to Dominique Legendre for + the report. + - Fix document about KEEP_PKEY_NAMES that also affect unique key and not + only primary key as it was specified in the documentation. Thanks to + Dominique Legendre for the report. + - Add tables generated by statistics on spatial index (MDXT_.*) into the + internal exclusion list. This join the already excluded table generated + by partition logging (USLOG$_.*) and materialized view logs (MLOG$_.*, + RUPD$_.*) + - Add DEFAULT_SRID configuration direction to permit change of the internal + default EPSG srid 4326. + - Fix new line after search_path settings. Thanks to Dominique Legendre for + the report. + - Triggers are now all excluded/allowed following the table names specified + in the ALLOW and EXCLUDED directive, no more on there own name which had + little interest. Thanks to Dominique Legendre for the feature request. + - Add support to COPY export with Spatial objects. Thanks to Legendre + Dominique for the great help to solve this problem. + - Fix default SRID value when a NULL value is returned by Oracle, SRID 8307 + and the corresponding EPSG SRID 4326. + - Update documentation on relation between PARALLEL_TABLES and FILE_PER_TABLE + - Add the -P or --parallel command line options and update documentation + about parallel table processing. + - Add PARALLEL_TABLES configuration directive to force ora2Pg to use on + process and one connection per table up to the number of CPU specified. + Thanks to menardorama for the feature request. + - Add PARALLEL_TABLES configuration directive to force ora2Pg to use on + process and one connection per table up to the number of CPU specified. + Thanks to menardorama for the feature request. + - Add --init_project and --project_base command line options to create a + migration template with a complete project tree, a generic configuration + file and script to automate export of all object in the project tree. + - Fix unwanted space before AND returned by limit_to_tables(). Thanks to + Alex Wang for the report. + - Add note about regex inclusion/exclusion not working with 8i database in + documentation + - Fix regex inclusion/exlusion of table that was not more working since the + inclusion of limit_to_tables() function. Thanks to alex wang for the patch + - Exclude dropped tables (those who are in the recycle bin) from export. + - When USER_GRANTS is disabled, aka login as dba user, force table list to + be checked against DBA_SEGMENTS with SEGMENT_TYPE of type table or table + partition. This could help solving some incomprehensible object found in + Oracle view ALL_TABLES. + - Fix query to retrieved list of tables, owner selection was set two time. + - Add support to automatic nested table export (TYPE+TABLE+COPY). + - Fix wrong export of materialized view log table. Thanks to Ronson Blossom + for the report. + - Update the SYSUSER array to exclude objects owned par those more users. + - Fix unwanted export of overflow table of an index-organized table. Thanks + to Ronson Blossom for the report. + - Update the SYSUSER array to exclude objects owned par those users. + - Display table owner in debug mode for SHOW_TABLE or SHOW_COLUMN. + - Add a section to give hint about converting Oracle outer join syntax to + ANSI. Thanks to Sven Medin for the links. + - Fix issue #82 again. Thanks to Sven Medin fro the report. + - Add first support to user defined queries to extract data from Oracle. + This feature add a new configuration directive named REPLACE_QUERY. + - Change program title when dump to file. + - Fix MODIFY_TYPE directive that was broken when using type with space + character. Thanks to Dmitry K. for the patch. + - Show missing view name in debug mode when exporting some views as table. + - Rewrite replace(a,b) with three arguments replace(a,b,'') for PostgreSQL. + Thanks to Dominique Legendre for the report. + - Convert all x <> NULL or x != NULL clauses to x IS NOT NULL. All x = NULL + are converted into x IS NULL. Thanks to Dominique Legendre for the report. + - Add warning at exit to signal when a OOM occurs. In that case, when a child + Ora2Pg process was silently killed by the OOM killer there was no information + that a failure occurs. + +2014 06 02 - v13.0 + +This major release adds first support to export Oracle Spatial Objects to PostGis +Spatial objects. There's also a new configuration directive to allow logging of +statement failures to prevent Ora2Pg to abort and continue to load valid data. +The other main feature is the possibility to convert DDL files without needing an +Oracle database connection, until now this was reserved to files containing stored +procedures. There's also several bug fixes. + + - Allow error logging during data import. This feature controlled by the + LOG_ON_ERROR directive allow you to not abort the data import process + when an error is encountered and to log to a file the COPY or INSERT + statement that generate the error. After fixing the statement you will + be able to load the missing data. Thanks to menardoram for the feature + request. + - Force export type to be INSERT when COPY is used and a table have a + GEOMETRY column. I can not find a solution to export as copy statement + for the moment. Thanks to Dominique Legendre and Vincent Picavet for + the help. + - Fix export of user defined type as object. Thanks to Shanshan Wang for + the report. + - Limit look up of objects to the ALLOW or EXCLUDE filter into the SQL + query instead of the Perl code to avoid retrieving huge list of objects + on such database. Thanks to menardorama for the feature request. + - Add support to spatial data export in INSERT mode. Still need some work + in COPY export mode if possible. + - Fix query to retrieve SRID that broken with patch on CONVERT_SRID. + - Fix wrong filter with ALLOW directive when getting list of partition. + - Add GRANT export read from an input file. + - Fix data type conversion when using input file and data type such + varchar2(10 BYTE). + - Add export of comment with TABLE and VIEW exports using an input file. + - Add extraction of TABLESPACE from an input file. + - Add support to SEQUENCE extraction from input file. + - Fix wrong filter with ALLOW directive when exporting partition. The + filter was done on partition name instead of table name, that mean + that setting ALLOW directive was resulting in no export at all. Thanks + to menardorama for the report. + - Add CONVERT_SRID configuration directive to control the automatic + conversion of SRID to standard EPSG using the Oracle SDO function + sdo_cs.map_oracle_srid_to_epsg() Oracle function. Thanks to Dominique + Legendre for the help. + - Fix a typo in the create index prefix on partitioned tables. Thanks + to menardorama for the patch. + - Fix non replacement of destination during SHOW_COLUMN and COPY export. + Using MODIFY_TYPE was only working in TABLE export. + - Force pl/sql conversion with TABLE export to replace advanced default + values. Fix code TRUNC(SYSDATE, MONTH) in default value and everywhere + that should be: date_trunc(month,LOCALTIMESTAMP). Thanks to menardorama + for the report. + - Fix code regarding unique partition index naming. Thanks to menardorama + for the report. + - Add PREFIX_PARTITION configuration directive. When enabled it will force + renaming all partition table name with the name of the parent table. + Thanks to menardoram for the feature request. + - Add AUTODETECT_SPATIAL_TYPE in configuration file and documentation + about this new directive. + - Add export of SDO_GEOMETRY column type. They are basically exported to + the non-constrained "geometry" type with SRID if defined. When the + configuration directive AUTODETECT_SPATIAL_TYPE is enable, Ora2Pg will + try to autodetect the geometry type, the dimension and the SRID used + to set a constrained geometry type. For example, in the first case + column shape with Oracle type SDO_GEOMETRY will be converted as: + + shape geometry(GEOMETRY) or shape geometry(GEOMETRY, 4326) + + and in the second case, with constrained geometry type: + + shape geometry(POLIGONZ, 4326) + + with a three dimensional polygon. Thanks to Vincent Picavet for the + feature request and specification. + - Add support to spatial index read from file. + - Add export of Oracle spatial index. For example, index: + CREATE INDEX cola_spatial_idx ON cola_markets(shape) INDEXTYPE IS MDSYS.SPATIAL_INDEX; + will be exported as + CREATE INDEX cola_spatial_idx ON cola_markets USING GIST(shape); + Thanks to Vincent Picavet / Oslandia for the feature request and explanations. + - Allow TRIGGER export to parse an input file with Oracle DML orders. + - Add PG_SUPPORTS_CHECKOPTION configuration directive to not remove + WITH CHECK OPTION in create view statement. It is supported in + PostgreSQL 9.4. + - Allow VIEW export to parse an input file with Oracle DML orders. + - Allow TABLE export to parse an input file with Oracle DML orders. + - Add SYNCHRONOUS_COMMIT configuration directive disabled by default. + This is the current behavior of Ora2Pg, it set synchronous_commit + to off before data import to PostgreSQL. This is only used when you + load data directly to PostgreSQL, the default is off to disable + synchronous commit to gain speed at writing data. Some modified or + old version of PostgreSQL, like Greenplum, do not have this setting. + - Add some useful information for Windows user in documentation. Thanks + to Roger Park for the report. + - Fix case when parentheses are omitted in index creation. Thanks to + Yuri Ushakov for the report. + - Fix export type PACKAGE when ALLOW is defined to extract only some + packages. Thanks to Maciej Bak for the report. + - Fix INSERT export where backslash should be escaped and single be + doubled in standard conforming string notation. Thanks to Yuri + Ushakov for the report. + - Add important note about LONGREADLEN and DATA_LIMIT that could need + to be adjusted to avoid out of memory. Thanks to Mike Kienenberger + for the patch. + - Fix case sensitivity issue with export of comment on column. Thanks + to Pierre Crumeyrolle for the report. + - Fix export of RAW data in COPY mode, was missing a backslash. Thanks + to jwiechmann for the report. + - Fix RAW data export in COPY and INSERT mode, RAW data type is returned + in hex by DBD::Oracle. Thanks to jwiechmann for the report. + - Fix one release 8i condition. + - Fix inexistent column USE_NO_INDEX with Oracle 8i and MVIEW export. + - Enclose call to utf8::encode and utf8::valid into eval. + - Fix export of constraint with Oracle 8i release. + - Fix unrecognized fatal error with 8i database. Thanks to UnvorherSeba + for the patch. + - Revert change level of error from fatal to error, when querying + materialized view. + - Change level of error from fatal to error, when querying materialized + view. + +2014 01 28 - v12.1 + +This is a maintenance release with some minor bug fixes and a new configuration +directive, INDEXES_SUFFIX, to allow appending a suffix to indexes names. + + - Fix example given for the WHERE configuration directive. Thanks to + Bob Treumann for the report. + - Add INDEXES_SUFFIX configuration option to allow append a suffix to + indexes names. + - Replace special charater ^M by \r as they are not supported by git. + - Fix IF EXISTS in alter table of sub _drop_foreign_keys. Thanks to + Francis Corriveau for the patch. + - Fix isolation level when exporting data. Thanks to Ludovic Penet for + the report. + - Fix regression when ora2pg tries to create foreign keys on tables or + to tables that are not selected for export. Thanks to Ludovic Penet. + - Add information about backslashed comma into directive MODIFY_TYPE + into Makefile.PL. + - Add missing MODIFY_TYPE definition in documentation. + - Allow backslashed comma into MODIFY_TYPE type redefinition. Example: + TABLE1:COL3:decimal(9\,6),TABLE1:COL4:decimal(9\,6). + Thanks to Mike Kienenberger for the report + - Fix missing single cote into create_materialized_view() call. Thanks + to Jacky Rigoreau for the patch. + - Fix some typo in documentation, thanks to Mike Kienenberger for the + report. + - Add a chapter about installing DBD::Oracle into documentation. Thanks + to Raghavendra for the patch. + - Fix case sensitivity on external table name with FDW export type. + Thanks to Guillaume Lelarge for the report. + - Fix export of materialized views when PG_SUPPORTS_MVIEW is disabled. + Thanks to Christian Bjornbak for the report. + - Update copyright. + +2013 10 22 - v12.0 + +This release fixes lot of issues and three new features. Using REORDERING_COLUMNS +directive you will be able to reorder columns to minimized the footprint on disc, +so that more rows fit on a data page. The PG_SUPPORTS_MVIEW will allow you to +export materialized with native PostgreSQL 9.3 syntaxe. The USE_TABLESPACE variable +will allow you to export object using their original tablespace. + + - Skip constraints on system internal columns (sys_nc...$) from export. + - Fix missing output directory in generic psql file for data loading. + - Add missing progress bar during TYPE and PARTITION export type. + - Remove duplicated message in debug mode during Oracle reconnection. + - Allow file input with create type declaration to use ora2pg converter. + Unsupported syntax is signaled into the output file. + - Exclude MLOG$.* and RUPD$.* table from export. + - Prevent export of indexes and constraints during FDW export type. + - Fix wrong total number of sequences shown in progress bar. + - Remove warning when PG_DSN is define during a export type that do not + support direct import into PostgreSQL. + - Auto switch prefix from DBA to ALL when error 942 is returned when + looking at tables informations. A hint is also displayed to ask for + activating USER_GRANTS or connect using a user with DBA privilege. + - Add REORDERING_COLUMNS configuration directive to allow reordering + columns during the TABLE export. This could help to minimized the + footprint on disc, so that more rows fit on a data page. Thanks to + Christian Bjornbak for the feature request. + - Fix call to unblessed reference at disconnect when direct import to + pg is not used. Thanks to Christian Bjornbak for the report. + - Fix regression in drop/create foreign keys and index during data + export. Thanks to Christian Bjornbak for the report. + - Fix truncate table error with parallel and direct data copy. Thanks + to keymaper for the report. + - Fix several other issues with parallel and direct data import. + - Fix trigger export on multi files when FILE_PER_FUNCTION is enabled. + - Fix issue on converting boolean values with non default values. + Thanks to Christian Bjornbak for the report. + - Fix boolean value for disabled key in default %BOOLEAN_MAP key/value. + - Fix case where INTO was wrongly replaced by INTO STRICT. Thanks to + Jacky Rigoreau for the report. + - Fix case where label after a END was not removed. Thanks to Jacky + Rigoreau for the report. + - Fix discard of input file parsing. Fix PERFORM replacement in PL/SQL + code wirh cursor. Thanks to Jacky Rigoreau for the report. + - Enable PG_SUPPORTS_MVIEW by default and update documentation. + - Replace DBA_DATA_FILES by USER_SEGMENTS to get database size to avoid + error ORA-00942. Thanks to Pierre Boizot for the report. + - Fix trigger conversion error. Thanks to Pierre Boizot for the report. + - Add support to PostgreSQL 9.3 materialized view syntaxe, this need a + new configuration directive PG_SUPPORTS_MVIEW to be enabled. + - Update default configuration file and documentation about USE_TABLESPACE. + - Add USE_TABLESPACE configuration directive to force ora2pg to use Oracle + tablespace name with table, constraints indexes and indexes if tablespace + in not in the default (TEMP, USERS, SYSTEM). Thanks to Rob Moolhuijsen + for the feature request. + - Allow DEFER_FKEY, when enabled during TABLE export, to create all foreign + keys as DEFERRABLE and INITIALLY DEFERRED. Thanks to David Greco for the patch. + - Fix non working ON_ERROR_STOP set to 0 during data export. + - Lot of code changes to fix dump to file in multiprocess mode. Ora2Pg will + also only drop/create constraints and indexes related to the allow/exclude + tables, thanks to Maciej Bak for the report. + - Force decimal character from Oracle output to be a dot. Thanks to Maciej Bak + for the report. + - Add default exclusion of Oracle recycle bin objects with name begining by BIN$. + - Fix escaping quote in table and column comments. Thanks to realyota for the report. + - Reduce DECODE migration cost from 2 to 1 unit. + - Reduce OUTER JOIN (+) migration cost from 3 to 1 unit. + - Add Time::HiRes to the requirement chapter for Perl <= 5.8. Thanks to + Mike Kienenberger for the report. + - Replace wrong use of --config instead of --conf into the documentation. Thanks + to Mike Kienenberger for the report. + - Fix regex used to rewrite CREATE VIEW code. Thanks to David Greco for + the patch. + - Fix an issue with oracle copies when primary key was negative. Thanks + to David Greco for the patch. + - Fix case sensitivity with SEQUENCE when preserve_case is enabled. + Thanks to Jean-Max Reymond for the report. + - Fix table COMMENT export when preserve_case is enabled. Thanks to + Jean-Max Reymond for the report. + +2013 05 28 - v11.4 + +This release fixes others several major issues on migration cost assessment that +was not addressed in previous release, please upgrade. + + - Fix other major issues in migration cost assessment. + - Redefine some migration cost values to be more precise. + +2013 05 27 - v11.3 + +This release fixes several major issues on migration cost assessment, especialy +with stored procedures with lot of lines or if you have lot of comments in that +code. You may want to run your database evaluation again as the estimated times +can be up to tree time lower on huge PL/SQL code. + + - Add full details about PL/SQL evaluation by ora2pg when --estimate_cost + or ESTIMATE_COST is enable. This will display cost units per keywords + detected in the function/package code. + - Fix wrong cost unit assessment on PL/SQL code size, this bug generated + very high migration cost assessment for functions/packages with lot of + lines. Please run your tests again, estimated times can be up to tree + time lower on huge code. + - Remove comments before code evalution. + - Fix file input parser for PL/SQL packages export when IS or AS was in + the next line than the CREATE PACKAGE BODY ... + - Exclude NOT NULL constraint from the count of CHECK constraints into + the TABLE report. + - Fix decimal precision in table migration assessment cost. + - Fix typo in changelog. + +2013 05 01 - v11.2 + +This release fixes several major issues especially with direct import of data +into PostgreSQL and Windows port that was both broken. + + - Update doc about Windows multiprocess issues and acknowledgements. + - Fix Windows OS issues using multiprocessing options by disabling + multiprocess support on this plateform. When -J or -j will be used a + warning will be displayed and Ora2Pg will simply run single process + like in previous 10.x versions. Thanks to Jean Marc Yao Adingra for + the report. + - Fix RAW and LONG RAW export to ByteA. Thanks to Prabhat Tripathi for + the report and testing. + - Fix patch regression on multiple TRUNCATE call for a single table. + Thanks to David Greco for the report. + - Placed calls to DB handle InactiveDestroy outside the forked process + to prevent fatal errors on Windows. Thanks to Jean Marc Adingra for + the report. + - Forked running processes are renamed into more readable name like + "ora2pg logger" for the progress bar, "ora2pg - querying Oracle" when + used with -J option and "ora2pg - sending to PostgreSQL" to better + know what is the current job of the process. + - Removed the use of /Y flag in Windows install script, this was causing + error "dmake: Error code 130, while making install_all". Thanks to + Jean-Marc Adingra for the report. + - Fix direct import to PostgreSQL that was just producing nothing. Thank + to David Greco for the patch. + - Fix ora2pg usage documentation. + - Add an underscore to CLIENT ENCODING in SHOW_ENCODING output to be the + same as the configuration directive. + +UPGRADE: please reinstall all as most of the files have changed. + +2013 04 07 - v11.1 + +This release adds partition data speed improvement by exporting data directly +from and into the destination partitioned table. There's also some bug fix on +RAW or LONG RAW data export and PL/SQL to PL/PGSQL code rewrite. + + - Adjust cost assessment for indexes, tables and tables partition. + - Add comment to report of index partition about local index only. + - Fix position of TRUNCATE TABLE in output file. + - Fix export of data from RAW or LONG RAW columns, they was exported + as hex string. Now data are converted using utl_raw.cast_to_varchar2() + function before being escaped for insert into a bytea. Thanks to Alex + Delianis for the report. + - Fix issue with Oracle TIMESTAMP(0) data export that add a single + ending point, ex: "2008-08-09 00:00:00.", this ending character is + now removed by format_data_type(). Thanks to Pierre-Marie Petit for + the report. + - Fix typo on MODIFY_STRUCT description. + - Force DEBUG to off in default configuration file. + - Change range PARTITION operators in the check conditions, >= and < + replaced by > and <=, corresponding to Oracle VALUES LESS THAN. + - Add ALLOW_PARTITION to limit data export to a list of partition name. + - PLSQL: Fix wrong replacement of SELECT by PERFORM during VIEW export. + - Partitioned tables data is now imported directly into the destination + tables instead of inserted into the main table and dispatched by the + trigger. Ora2Pg will automatically detect the in/out table partition, + there's nothing to configure. + - PL/SQL: Do not allow decode() rewrite by case/when/else when there + is a function call in it. + - Fix Error when Compress::Zlib is not installed, this module is not + mandatory. + +UPGRADE: please reinstall all as all files have changed. + +2013 03 24 - v11.0 + +This is a new major release because it adds support to multiprocessing to export +data in parallel mode, this allow to improve speed during data import by more +than ten times. This multiprocessiing capabilities allow Ora2Pg to be closer than +the speed of any ETL. To compare speed or allow using Kettle for data import, +there's now a new export type to obtain Kettle XML transformation files. This +release adds also lot of work on speed improvement to scan Oracle database with +huge number of object. + + - Add documentation about JOBS, ORACLE_COPIES, DEFINED_PK configuration + directive and informations about KETTLE export type. + - Add KETTLE export type to generate XML transformation file definition + for Penthatlo Data Integrator (Kettle). Thanks to Marc Cousin for the + work. Example of use: + ora2pg -c ora2pg.conf -t KETTLE -j 12 -J 4 -o loaddata.sh + - Fix major bug in export of auto generated named constraint. Thanks to + mrojasaquino fot the report. + - Show number of rows in the top largest tables. + - Add TOP_MAX description to the documentation. + - Add the TOP_MAX directive to default configuration file and update + documentation. Directive used to control the top N tables to show. + - Add top N of largest tables in SHOW_TABLE, SHOW_COLUMN and SHOW_REPORT + export type. + - Fix progressbar output when ora2pg is interrupted by ctrl+c. + - Add JOBS, ORACLE_COPIES and DEFINED_PK directives to configuration file. + JOBS replacing THREAD_COUNT but backward compatibility is preserve. + - Add 3 new command line options, -j | --jobs and -J | --copies, used to + set the number of connection to PostgreSQL and Oracle for parallel + processing. The third, -L | --limit is used to change DATA_LIMIT at + command line. + - Add multiprocess support on data export. With the help of Thomas Ogrisegg. + - Add more schema in SYSUSERS that should not be exported. + - Add full detailed information about SYNONYM in SHOW_REPORT. + - Add MODIFY_TYPE configuration directive to allow some table/column + type to be changed on PostgreSQL side during the export. + - Fix objects type count in progressbar of SHOW_REPORT. + - Restrict table and index in SHOW_REPORT to the tables defined in ALLOW + and EXCLUDE directives. + - Show total number of rows in SHOW_TABLE and SHOW_REPORT output. + - Add top 10 of tables sorted by number of rows in SHOW_TABLE and + SHOW_REPORT output. + - Fix typo in SYNONYM objects. + - Add report of top ten tables ordered y number of rows. + - Rewrite most of the Oracle schema storage information extraction for + speed improvement. + - Use Hash to store column informations. + - Fix %unique_keys declaration in _table() method. + - Remove call to _table_info() from SHOW_REPORT code as those informations + are already loaded with the _table() method. + - Fix missing column definition on TABLE export. + - Add progress bar during output generation following export type. + - Add STOP_ON_ERROR configuration directive to enable/disable the call to + ON_ERROR_STOP into generated SQL scripts. Thanks to Ludovic Penet for + the feature request. + - Huge speed improvement on columns informations retrieving. + - Fix progress bar to keep the total number of tables related to the ALLOW + or EXCLUDE configuration directives. Thanks to Ludovic Penet for the report. + - Change return type of function _table_info(), it now returns data instead + of the database handle. + - Improve speed on indexes and constraints extraction for database with huge + number of tables. + - Improve performance to retrieve columns information and comments. + - Remove report of column details during export in debug mode, use SHOW_COLUMN + instead. + - Remove call to upper() in objects owner condition to improve performance + with database with huge number of objects. + - Add a fix to not export foreign key for exclude tables. Thanks to Ludovic + Penet for the report. + - Fix Windows install issue with copying ora2pg.conf.dist. Thanks to + Dominique Fourdrinoy for the report. + - Increase the cost of Oracle function not converted to PG automatically. + +UPGRADE: reinstall all is required to override the old installation, you may use the +new ora2pg.conf.dist file which included the new configuration directives. + +2013 01 15 - v10.1 + +This release adds HTML report for migration cost assessment and some bug fix. + + - Fix global where should not be overwritten. Thanks to Dan Harbin for + the patch. + - Fix bug/typo in boolean replacement, where a colon was used instead + of a single quote. Thanks to Alex Delianis for the patch. + - Update copyright. + - Add detection of additional Oracle functions for better migration + cost assessment. + - Update documentation. + - Force report detail in lowercase. + - Added information about the migration cost value to the reports. + - Add --dump_as_html command line option and DUMP_AS_HTML configuration + directive. + - Allow migration report to be generated as HTML. + - Separate report generation code from data collection code. + +2012 12 12 - v10.0 + +This is the first version of Ora2Pg 10.x series, that is a major release. +Overall numerous improvements and bugs fixes there's now a new export type: +SHOW_REPORT that will output a report of all objects contained in your Oracle +database and some comments on how they will be exported. With this report you +can use a new directive ESTIMATE_COST to ask to Ora2Pg to evaluate the database +migration cost in terms of man days. There's also an other new configuration +directive EXTERNAL_TO_FDW, disable by default, to permit the export of all +Oracle external tables as file_fdw foreign tables. + +The database content report and the migration cost estimation is a work in +progress so all feedback on these new features are welcome. Here is the complete +changelog: + + - Update documentation about ora2pg usage and new feature. + - Fix quote escaping on table comments. Thanks to Sebastian Fischer. + - Fix some other issues with 8i databases, added database version auto- + detection to avoid printinf warning. Thanks to Sebastian Fischer for + the help. + - Allow null value in BFILE to the oar2pg_get_bfilename(). + - Update documentation about BFILE export. + - Add drop function ora2pg_get_bfilename() when necessary. + - Add support to BFILE external path export by creating a function + ora2pg_get_bfilename( p_bfile IN BFILE ) to retrieve path from BFILE. + BFILE will be exported as text field with the full path to the file as + value. Note that this is the first time that Ora2Pg need write access + to the Oracle database, if you do not have BFILE or you have set the + corresponding PostgreSQL type asd bytea (the default) the function + will not be created. + - Fix a performance issue when extracting BLOB with a LongReadLen upper + than 1MB. + - Fix priviledge on schema created from Oracle package body. Thanks to + Dominique Legendre for the report. + - Add object type in comment before priviledge extraction. + - Order output of grant to groups grants by object types. This is useful + to quickly disable some SQL orders corresponding of not already loaded + objects. Thanks to Dominique Legendre for the feature request. + - Fix progress bar output. + - Fix priviledge on sequence, tablespace and schema. + - Fix backward compatibility with Oracle 8i, remove query with JOIN. + Thanks to Sebastian Fischer for the report. + - Fix backward compatibility with Oracle 8i on priviledge extraction. + Thanks to Sebastian Fischer for the report. + - Fix backward compatibility with Oracle 8i on index extraction. Thanks + to Sebastian Fischer for the report. + - Add more precision in cost estimation. + - Add somme other PL/SQL uncovered code detection. + - Add more debug information during data extraction. + - Removed progress bar when debug is enabled. + - Add report and estimate cost about CHECK constraint and function + based indexes. + - Update documentation about new export directives SHOW_REPORT and + ESTIMATE_COST. + - Add --estimate_cost and --cost_unit_value command line options. + - Add ESTIMATE_COST and COST_UNIT_VALUE to default configuration file. + - Rewritte and extend support to ROWNUM replacement. + - Remove incompatible grants between Oracle and the PortgreSQL export, + especially on views. + - Limit GRANT export to VALID object. Activate EXPORT_INVALID to enable + grants export on all object. + - Add export of VALID only views. To export all with INVALID ones you + must activate the EXPORT_INVALID directive. Thanks to Dominique + Legendre for the feature request. + - Fix issue in substr() pl/sql replacement, thanks to Dominique + Legendre for the report, plus add other code replacements in pl/sql. + - Fix issue with function name not on the same line as the create + statement - was affecting file input only. + - Add report of number of JOB object in the database (SHOW_REPORT). + - Add PL/SQL replacement of various form of EXEC function call. + - Remove creation of password with users that are not requiring + password. Thanks to Dominique Legendre for the feature request. + - A sql type and a precision can now be used in REPLACE_AS_BOOLEAN to + replace all filed with that type as a boolean, example: + NUMBER:1 will replace all field of type NUMBER(1) as a boolean. + - Fix grants on partition export, will now used all_ and user_ tables. + - Fix removing of newline in the DECLARE clause. Thanks to Dominique + Legendre for the report. + - PostgreSQL client_encoding is now forced to UTF8 when BINMODE is set + to utf8. Thanks to Dominique Legendre for the report. + - Replace DISABLE TRIGGER ALL by DISABLE TRIGGER USER following the value + if USER_GRANTS to avoid permission denied on constraint trigger when + data are load under a non PG superuser. Thanks to Dominique Legendre + for the report. + - Rename DISABLE_TABLE_TRIGGERS to DISABLE_TRIGGERS and set default value + to 0. Other values are USER or ALL following the connected user. + - Fix missing newline after comment in PL/SQL code. Thanks to Dominique + Legendre for the report. + - Fix report message on external table export. + - The export TYPE have been entirely rewritten to only export supported + user defined types. Exported are: Nested Tables, Object type, Type in + herited and Subtype, Varrays. Associative Arrays, Type Body and type + with member method are not supported. + - When FILE_PER_INDEX is enable, SQL order to move indexes in their + respective tablespace will be written into a dedicated file prefixed + by TBSP_INDEXES_. + - Fix location on external table export. Thanks to Thomas Reiss for + the help. + - PG_SUPPORTS_INSTEADOF is now activated by default, that mean that + migration should be done on PG >= 9.1. + - Remove obsolete --xtable commande line option, should be replaced by + --allow, backward compatibility is preserved. + - Add EXTERNAL_TO_FDW configuration directive, disable by default, to + export all Oracle external tables as file_fdw foreign tables. + - Fix an other case where user defined type were not exported with an + ending semi-colon. Thank to Dominique Legrendre for the report. + - Fix export of user defined type with extra ");" at end of the type + definition and remove system types from export. Thanks to Dominique + Legendre for the report. + - Add PLSQL replacemement of currval. Thanks to Thomas Reiss for the + patch. + - Add PLSQL replacement of PIPELINED and PIPE ROW by SETOF and RETURN + NEXT. + - Add rewrite of Oracle DETERMINISTIC function into PostgreSQL + IMMUTABLE function. + - Fix copy during install on MacOSx and add /Y option to windows + install copy to force overwrite existing files. Thanks to Dennis + Spaag for the report. + - Fix issue exporting rows with perl ARRAYS ref. Thanks to Sorin + Gheorghiu for the report. + - Add report of number of database link in SHOW_REPORT output. + - Fix major bug on export of NUMBER with precision, they was all + exported as bigint. Thanks to Dominique Legendre for the report. + - Add progress bar during SHOW_REPORT export. + - Add detailed report about index in SHOW_REPORT output. + - Fix data export when schema was given in lower case. Thanks to + Dominique Legendre for the report. + - Add SHOW_REPORT export type to display a full summary of the Oracle + database content. + - PLPGSQL: add the name keyword to XMLELEMENT call. Thanks to Thomas + Reiss for the hint. + - Add SHOW_VERSION export type to display the version of Oracle. + - Add COLLATION to the keyword list. Thanks to Dominique Legendre for + the report + - Change documentation to add more detail on exporting Oracle views as + PostgreSQL tables based on the new VIEW_AS_TABLE directive. + - Add -a | --allow option and --view_as_table to ora2pg script. + - Add VIEW_AS_TABLE configuration option to allow export of view as + table and permit the additional use of the ALLOW or/and EXCLUDE + directive. Thanks to Dominique Legendre for the feature request. + - Removed conflict with transaction when DEFER_FKEY was enabled and + allow DEFER_FKEY and DROP_FKEY to be enabled both. Before, only + DEFER_FKEY was used in this case, now both are used and of course + DEFER_FKEY is wasted. Thanks to Dominique Legendre for the report. + - Directives ALLOW and EXCLUDE are now usable with all kind of object + following the export type. + - Rename TABLES directive as ALLOW to be less confusing, backward + compatibility is preserved. + - Thanks to Dominique Legendre for the feature request. + - Remove auto ordering of table export following the foreign keys to + fix an infinite loop. Thanks to Siva Janamanchi for the report. + - Rewrite the view as table export to reuse the same code as table + export, old code was resulting in issues with disable triggers and + deferring constraints. + - Remove alter session to set NLS_NCHAR that was returning error on some + installation. Thanks to Dominique Legendre for the report. + - Fix replacement of IS SELECT wrongly replaced by IS PERFORM in some + case. Thanks to Dominique Legendre fot the report. + +UPGRADE: Almost all files have changed so a new installation is required. + +2012 10 07 - v9.3 + + - Add auto detection of Oracle character set and the corresponding + PostgreSQL client encoding to use. NLS_LANG and CLIENT_ENCODING + configuration directives can be leaved commented, Ora2Pg will set + their values automatically. + - Add PL/SQL replacement of CURSOR IS SELECT by CURSOR FOR SELECT and + IS REF CURSOR by REFCURSOR. Thanks to Dominique Legendre for the + report. + - Fix missing set client_encoding orders into fonction or procedure + export file. Thanks to Dominique Legendre for the report. + - Fix not working SKIP configuration directive. Thanks to Siva + Janamanchi for the report. + - Add configuration directive NULL_EQUAL_EMPTY to disable the IS NULL + and IS NOT NULL replacement into PL/SQL code. Enabled by default. + Thanks to Dominique Legendre for the feature request. + - Remove exclusion of object names with the dollar sign. Thanks to + Konrad Beiske for the suggestion. + - Fix timestamp with time zone when microsecond is enabled. Thanks + to Siva Janamanchi for the report. + - Fix extra semi-column when PKEY_IN_CREATE is enabled. Thanks to + Siva Janamanchi for the report. + - Update configuration about boolean replacement. + - Allow any column type replacement as a boolean in PostgreSQL, values + will be converted as well. Thanks to Konrad Beiske for the feature + request. + - Add REPLACE_AS_BOOLEAN and BOOLEAN_VALUES configuration directives + to allow type replacement with a boolean. Thanks to Konrad Beiske + for the feature request. + - Add new configuration directive PKEY_IN_CREATE to add primary keys + definition in CREATE TABLE statement instead of creating them after + with an ALTER TABLE statement. For Greenplum database, primary key + must be created with the CREATE TABLE statement so you may want to + enable this configuration directive. Thanks to Siva Janamanchi for + the feature request. + - Add new configuration directive USE_RESERVED_WORDS to force Ora2Pg to + auto-detect PostgreSQL reserved words in Oracle object's names and + automatically double quote them. Thanks to Siva Janamanchi for the + feature request. + - SHOW_TABLE and SHOW_COLUMN will now display a warning when Oracle + object's name is a PG reserved words. Those names will need to be + renamed or double-quoted (see USE_RESERVED_WORDS). + - Add TIMESTAMP WITH LOCAL TIME ZONE Oracle type conversion to timestamp + and force timestamp with time zone format to use TZH:TZM. Thanks to + Siva Janamanchi for the report. + - Fix table and column replacement issues introduced with path that + removed double-quote when PRESERVE_CASE is disabled. Thanks to Steve + DeLong for the report. + - PLPGSQL convertion: Fix SELECT replacement with PERFORM in VIEW + declaration. Thanks to Thierry Capitaine for the report. + - Add display Ora2Pg type conversion map between Oracle originals types + and PostgreSQL's types when using export type SHOW_COLUMN. Thanks to + Thierry Capitaine for the feature request. + - Reorder command line options in ora2pg script usage and documentation. + - Add call to quote_ident() and quote_literal() into materialized + functions to secure parameters. + - Fix major issue in pl/sql to pl/pgsql conversion with multiple package + declaration in the same code record. Thanks to Marc Cousin for the + report. + - Add data type TIMESTAMP WITH TIME ZONE. Thanks to Siva Janamanchi for + the report. + - Add new export type: MVIEW to allow export of all materialized views + as snapshot materialized view (fully reload of the view). + - Add -e | --exclude option to Perl script ora2pg to exclude some given + objects from the export. It will override any value of the EXCLUDE + directive. The value is a coma separated list of object name or regex. + - Update domumentation about the EXCLUDE directive change. + - Allow exclusion from export of functions, procedures and functions in + package by specifying a list of name or regex in EXCLUDE directive. + Thanks to Keith Fiske from Omniti for the feature request. + +UPGRADE: Almost all files have changed so a new installation is required. + +2012 09 05 - v9.2 + + - In plpgsql conversion, SELECT without INTO becomes PERFORM. + - In plpgsql conversion, EXECUTE IMMEDIATE replaced by EXECUTE. + - Fix DATA_TYPE value in configuration file. + - Fix case sensitivity on data export using COPY mode. + - Directive XML_PRETTY is now disabled by default as it is better to + use getClobVal() to get the XML data out of an xmltype column. + - Add documentation about regex usage int EXCLUDE and TABLES directives. + - Remove all double-quote around object name when CASE_SENSITIVY is + disabled. Thanks to Dominique Legendre for the suggestion. + - Rename CASE_SENSITIVE by PRESERVE_CASE to be less confusing, backward + compatibility preserved. Thanks to Dominique Legendre for the request. + - Add support to user defined type data export. Before it will simply + export an array reference ARRAY(0xa555fb8), now the array is explored + and inserted as ROW(col1,col2,...). Thanks to Mathieu Wingel for the + feature request. + - Fix bug in direct data import in postgresql with COPY: pg_putcopydata + can only be called directly after issuing a COPY FROM command. Thanks + to Steve Delong for the report. + - Add warning at any debug level before abort when a data file already + exist during data export. + - Fix issue with oracle sensitivity when exporting data. + - Fix search_path on package export, indexed and constraints files on + TABLE export. + - Remove obsolete ORA_SENSITIVE configuration directive, thanks to + Dominique Legendre it is no more used. + - Force automatic conversion of PL/SQL when entry is an input file. + - Fix errors in main file for package loader with FILE_PER_FUNCTION + enabled. + - Fix case where package body where not exported. + - Add missing EXPORT_INVALID directive into default configuration file. + - Fix replacement of END followed by the name of the function, the semi- + colon was removed. + - Fix case sensitivity issue in INDEX creation. + - Fix case sensitivity issue in CHECK constraint and a typo in a + progress bar variable. + - Replace old DATA export type by INSERT in configuration file. + - Fix case sensitivity issue in ALTER TABLE ... ADD CONSTRAINT. Thanks + to David Greco for the report. + - Add set client_encoding before table output to handle character + encoding into comments and possibly objects names. + - Fix some case sensitivity issue with schema name. Thanks to Dominique + Legendre for the report. + - Do not display warning message about direct import if no connection + to a PostgreSQL database is defined. + - Allow multiple export type to be specified into the ora2pg -t command + line option. + - Dump progress bar to stderr instead of stdout to separate logs. + - Add new -q | --quiet option to perl script ora2pg to disable progress + bar. + +2012 08 19 - 9.1 + + - Add progress bar to show data export progression. + - Add -q | --quiet option to ora2pg perl script to disable progress bar. + - Change documention about tnsnames.ora to mark it is not necessary. + - Add progress bar during data export, per table and globaly. + - Replace export type DATA by INSERT to mark the difference with COPY + and avoid confusion. Documentation is updated and full backward + compatibility preserved. + - Improve Oracle case sensitivity detection on column and update + documentation about ORA_SENSITIVE directive + - Direct import for COPY statement now used DBD::Pg and pg_putcopydata() + instead of a pipe to psql command. + - Fix case sentitivity issue on disabling/enabling all triggers. + - Add autodetection of case sensitivity with column name. + - Move trunc() to data_truc() convertion into the ALLOW_CODE_BREAK part. + - Update comment about FILE_PER_FUNCTION in configuration file. + - Fix NOT NULL constraint add twice, the first time in the column + definition and the second time in an ALTER TABLE ... ADD CONSTRAINT + ... CHECK ( ... NOT NULL). Reported by Dominique Legendre. + - Add support to direct CALL of stored procedures in trigger definition. + Reported by Dominique Legendre. + - Remove index creation on primary and unique key autogenerated by + PostgreSQL. + - Fix PL/SQL to PLPGSQL automatic convertion on index when exporting + data with DROP_INDEX activated. + - Fix DROP_INDEX to only delete indexes that will be created at end. + - Fix search path when exporting data with EXPORT_SCHEMA disabled + - Add missing documentation about the LOGFILE directive + - Fix case sensitivity on sequence export. They will now always be + insensitive as in PostgreSQL its called is converted between quotes: + nextval('seq_name'). Reported by Dominique Legendre. + - Limit export of primary and unique key if KEEP_PKEY_NAMES is enabled + to those who are not autogenerated by Oracle. Reported by Dominique + Legendre. + - Trigger export is now limited to those belonging to table that are not + excluded from the export (see TABLES and EXCLUDE directives). Reported + by Dominique Legendre + - Fix case sensitivity on trigger export. + - Fix data export failure in table with column name with accent. + Reported by Dominique Legendre. + - Fix set client_encoding syntax. Reported by Dominique Legendre + - Add automatic try with oracle sensitivity when an error occurs during + retreving table information. This additionaly also fixes an error when + table has accent on his name. + - Fix replacement of user defined data type with directive DATA_TYPE. + Reported by Dominique Legendre. + - Fix function or procedure detection with external input file. Reported + by Arul Shaji. + - Update documentation about Windows installation and ActiveState Perl + distribution. Thanks to Stephan Hilb for the report. + - Fix date format issue by forcing NLS_DATE_FORMAT to format: + YYYY-MM-DD HH24:MI:SS. Thanks to Stephan Hilb for the report. + - Remove obsolete pod documentation in Ora2Pg.pm. + - Add new configuration directive CREATE_SCHEMA to disable the sql + command of schema creation at top of the output file during TABLE + export type. Patch by David Greco. + - Added converting INSERTING/UPDATING/DELETING to PG_OP=INSERT, etc. + Patch by David Greco. + - Fix parsing leading ':' on triggers, as they generally have :NEW and + :OLD to denote the new and old recordsets. Patch by David Greco + - Add new PG_INTEGER_TYPE configuration directive activated by default, + to limit conversion into postgresql integer or bigint of Oracle number + without scale - NUMBER(p), PG_NUMERIC_TYPE is now reserved to convert + NUMBER(p,s). Patch by David Greco. + - Limit numeric with precision <= 9 to be converted as integer, numeric + with precision >= 10 will be converted to bigint to handle integer + above 2147483647. Patch by David Greco. + - Add plsql to plpgsql automatic conversion on check constraints. Patch + by David Greco. + - Add plpgsql replacement, patch by David Greco: + REGEX_LIKE( string, pattern ) => string ~ pattern + - Update documentation about NOESCAPE and STANDARD_CONFORMING_STRING + - Change place of the ENABLE_MICROSECOND into the documentation. + - Fix forgot to add documentation about encryption with Oracle server. + - Add missing DISABLE_COMMENT configuraton directive in default + configuration file and update documentation + +2012 07 15 - 9.0 + + - Remove call to obsolete BINDIR and MANDIR variables into packaging + scripts to reflect the changes in Makefile.PL. + - Update documentation about installation of Ora2Pg under Windows. + - Automatically set LONGREADLEN to ORA_PIECE_SIZE length if the last one + is larger, for CLOB export. + - Change Makefile.PL and source tree to fully support installation under + Windows OSes. + - Change double quote by single in Makefile.PL perl replacement call. + - Replace double quote by single one in $CONFIG_FILE default setting to + simplify automatic replacement at install. + - Fix CLOB export that was limited to 64Kb even with LONGREADLEN defined + to an upper value. Patch use the ora_piece_size DBD::Oracle prepare + attribute. Patch by Mohamed Gargouri. See here for more detail: + http://search.cpan.org/~pythian/DBD-Oracle-1.46/lib/DBD/Oracle.pm#Piecewise_Fetch_with_Polling + - Add a note into documentation about encrypted communication between + Ora2Pg and Oracle. Note by Jenny Palomino. + - Change documentation to reflect change to the format of the Oracle + timestamp with millisecond. This format is now enabled by default in + configuration file. + - Fix bug with LONGREADLEN and LONGTRUNCOK when exporting LOB that was + not applied even after change into the configuration file. Reported + by Mohamed Gargouri + - Fix microsecond format FF3 not compatible with Oracle 8i. Set to FF. + - Add a warning to stderr when a table export need that ORA_SENSITIVE + be enabled. + - Fix case where Oracle indexes with same name as a constraint was not + exported - Rodrigo + + The following are old patches that was not applied to v8.10 and the git + repository: + + - Fix creation of bad constraint for each indexes. + - Add DISABLE_COMMENT configuration directive to remove comments from + TABLE export type. Comments are exported by default. + - Fix a bug in removing function name repetion at end + - Add PL/SQL to PLGPSQL replacement of the to_number function + - Fix PL/SQL to PLGPSQL replacement of substr into substring + - Add replacement of specials IEEE 754 values BINARY_(FLOAT|DOUBLE)_NAN + and BINARY_(FLOAT|DOUBLE)_INFINITY by NaN and Infinity on PLPGSQL + conversion and on data export - Thanks to Daniel Lyons. + - Fix return type of function with just OUT or INOUT params. Thanks to + Krasi Zlatev for the patch. + - Add schema name on functions or procedures export when EXPORT_SCHEMA + is activated. Thanks to Krasi Zlatev for the patch. + - Fix case sensitivity issue with schema on partition export. + - Fix case sensitivity issue with --xtable option. + - Fix issues with case sensitivity on the schema owner set into the + SCHEMA configuration directive. + - Add default search_path on schema for contraints, index and data + export when EXPORT_SCHEMA is activated. + - Fix case sensitivity issue in search_path. + - Force Oracle datetime format to be YYYY-MM-DD HH24:MI:SS.FF in client + session to prevent other defined output format. Thanks to Aaron Culich + for the patch. + - Add export/import of table and column comment. Thanks to Krasi Zlatev + for the patch. + +2012 06 26 - 8.13 + + - Fix broken export with missing single quote in Oracle timestamp export + formating with to_char(timestampcolumn, YYYY-MM-DD HH24:mi:ss). Thanks + to Steve Delong for the report. + +2012 06 22 - 8.12 + + - Add nex configuration directive ENABLE_MICROSECOND to allow timestamp + to be exported with millisecond precision. Thanks to Patrick King for + the feature request. + - Fix multiple quote on foreign keys column names. Thanks to Vitaliy for + the report. + - Add new export type FDW to allow table export as foreign table for + oracle_fdw. Thanks to David Fetter for the feature request. + - Fix typo in LongTruncOk variable name. Thanks to Magnus Hagander for + the patch. + - Add XML_PRETTY configuration directive to replace getStringVal() by + getClobVal() when extracting XML data. Thanks to Magnus Hagander for + the patch. + - Fix case sensitivity issue in ALTER TABLE and TRUNCATE statement. + Thanks to Magnus Hagander for the patch. + +UPGRADE: Ora2Pg.pm and ora2pg perl scripts have changed as well as configuration +file. Documentation has been updated too so you'd better install all again. + +2012 04 30 - 8.11 + + - Fix an error when running ora2pg directly against PG, index and + constraints are created against PG instead of being written to + the output file. Thanks to David Greco for the report. + - Ora2Pg will now output a warning message when direct import to PG + is set with other import type than COPY and DATA. + - Fix NUL character removing on LOB to bytea export. Thanks to info31 + on PostgresqlFr for the report. + +2012 03 11 - 8.10 + + - Add two configuration directives to control the BLOB export. + LONGREADLEN to set the database handle's 'LongReadLen' attribute + to a value that will be the larger than the expected size of the + LOB. LONGTRUNKOK to bypass the 'ORA-24345: A Truncation' error. + Thanks to Dirk Treger for the report. + - Fix install problem on non-threaded Perl and the threads package. + Replace use() by require() call. Thanks to Ian Sillitoe for the patch. + - Fix strange Oracle behaviour where binary_double infinity is exported + from Oracle as '~'. Replaced by 'inf'. Thanks to Daniel Lyons for the + report. + +UPGRADE: only Ora2Pg.pm have changed so you can just override it. See also +documentation for new configuration directives: LONGREADLEN and LONGTRUNKOK. + +2011 11 07 - 8.9 + + - Fix double quote into file name of package function export when case + sensitivity is preserved. + - Add support to XMLType data extraction. Thanks to Aaron Culich for + the report. Before this release, xml data was exported as a Perl + array reference. + - Fix bug during foreign key export when foreign keys have different + owners. Thanks to Krasi Zlatev for the patch. + - Add support to plpgsql conversion during index extraction as many + index use some Oracle function on their declaration. Thanks to + Sriram Chandrasekaran fot the feature request. + - PLSQL: Add replacement of Oracle subtr() by PostgreSQL substring(). + Thanks to Sriram Chandrasekaran fot the feature request. + - PLSQL: Add replacement of Oracle decode() by PostgreSQL CASE/THEN/ELSE. + Thanks to Sriram Chandrasekaran fot the feature request. + Note that this two replacement was not implemented because they could + break the code if there's complex subqueries inside their declaration. + This is why you can enable it by setting ALLOW_CODE_BREAK to 1 (new). + In later release this directive will be enable by default. + - Add output ordering on some object name so that results between two + runs can be compared. Thanks to Henk Enting for the patch. + - Fix misshandling of all cases of RAISE_APPLICATION_ERROR rewrite into + RAISE EXCEPTION concatenations. Thanks to Krasi Zlatev for the report. + +UPGRADE: only Ora2Pg.pm and Ora2Pg/PSQL.pm have changed so you can just override them +if you dont want to reinstall all. + +2011 10 13 - 8.8 + + - Before that release when you upgraded Ora2Pg using Makefile, the old + ora2pg.conf was renamed as ora2pg.old. This can lead to lost previous + configuration, the old ora2pg.conf is now untouched and the new one is + installed as ora2pg.conf.new. + - Renamed ora2pg.pl into ora2pg_pl in the package before installation + to avoid the copy of the perl script into the site Perl install dir. + It is still installed as ora2pg in /usr/local/bin by default. + - Fix errors that appeared to be due to no quoting on the field names + when ORA_SENSITIVE is enabled. Thank to Sam Nelson for the patch. + - Limit case sensitivity on check constraints to column names only, + before that if there was a value between double quote into the check + constraint, it was wrongly changed to lower case. + - Fix broken case sensitivity at data export when disabling/enabling + triggers and truncating tables with copy or insert statement. + - Change Ora2Pg version in packaging files that was still in 8.5. + +UPGRADE: only Ora2Pg.pm have changed so you can just override it. + +2011 09 07 - 8.7 + + - The escape_bytea() function has been rewritten using a prebuild array + to gain twice of performances. Thanks to Marc Cousin from Dalibo for + the patch. + - Improve speed of bulkload data by disabling autocommit by issuing a + BEGIN at the start and COMMIT at the end. + - Add multi-threading support. It is only used to do the escaping to + convert LOBs to byteas, as it is very cpu hungry. There's a lot of + CPU-waste here. The threads number is controlled by a new configuration + directive: THREAD_COUNT. Putting 6 threads will only triple your + throughput, if your machine has enough cores. If zero (default value), + do not use threads, do not waste CPU, but be slower with bytea. + Performance seems to peak at 5 threads, if you have enough cores, and + triples throughput on tables having LOB. Another important thing: + because of the way threading works in perl, threads consume a lot of + memory. Put a low (5000 for instance) DATA_LIMIT if you activate + threading. Many thanks to Marc Cousin for this great patch. + - Fix standard_conforming_string usage on export as INSERT statement. + - Fix an issue with importing Oracle NULL character (\0 or char(0)) with + bytea and character data with UTF8 encoding. Now whatever is the data + type or the encoding, this character is simply removed to prevent the + well known 'ERROR: invalid byte sequence for encoding "UTF8": 0x00.' + Thanks to Jean-Paul Argudo from Dalibo for the report. + - Fix an incorrect syntax for "for each statement" triggers. + Thanks to Henk Enting for the report. + - Add comment at end of line to signal on which cursor the replacement + on " EXIT WHEN (...)%NOTFOUND " is done. This will return something + like "IF NOT FOUND THEN EXIT; END IF; -- apply on $1". Thanks to + jehan Guillaume De Rorthais from Dalibo for the report this help a lot + during Pl/Pgsql code review. + - Fix table/column name replacement on constraint creation and dropping + when REPLACE_TABLES/REPLACE_COLS is set during DATA/COPY export. + - Fix table/column name replacement on indexes creation and dropping + when REPLACE_TABLES/REPLACE_COLS is set during DATA/COPY export. + - Remove unused table name parameter in _drop_indexes() function. + - Add support to REPLACE_TABLES/REPLACE_COLS during schema export. + Before this release those replacements were only applied to DATA or + COPY export. You can now use it in schema export, it will replace + table and/or column names in the TABLE/INDEX/CONSTRAINT schema export. + MODIFY_STRUCT is still limited to DATA or COPY export as it have no + sense outside this export. Unfortunately those replacements can not be + done easilly in other export type like TRIGGER, FUNCTION, etc. so you + must still edit this code by hand. + - Use the bundled Perl Config module to detect if Perl is compiled with + useithread. This mean that the old local defined %Config hash has been + replaced by %AConfig. + - SKIP indices is now obsolete and must be replaced with SKIP indexes. + backward compatibility is preserved. + - The file generated when FILE_PER_INDEX is activated has been renamed + into INDEXES_... instead of INDICES_... + - Add a warning on tablespace export when Oracle user is not a dba. + - Fix fatal error when dumping to one file per function with double + directory output. + - Fix double print of FATAL messages and dirty database disconnect on + fatal errors. + - Add setting of client_encoding into each export type as defined in + the configuration file. + - Update web site documentation. + +UPGRADE: Ora2Pg.pm, Ora2Pg/PGSQL.pm and ora2pg have changed so they must be +overwritten. There's also changes in the configuration file and documentation +has changed as well. Backward compatibility is fully preserved. + + +2011 07 07 - 8.6 + + - Remove "use strict" from head of Ora2Pg.pm that breaks view export. + This is usually removed before public release, but not this time. + Thanks to Jehan Guillaume de Rorthais from Dalibo for the report. + - Add a new configuration directive called COMPILE_SCHEMA that force + Oracle to compile the PL/SQL before code extraction to validate + code that was invalidate for any reason before. If you set it to 1, + you will force compiling of the user session schema, but you can + specify the name of the schema to compile as the value too. Thanks + to Jean-Paul Argudo from Dalibo for the solution. + - Add new configuration directive EXPORT_INVALID to allow export of all + PL/SQL code even if it is marked as invalid status. The 'VALID' or + 'INVALID' status applies to functions, procedures, packages and user + defined types. + - Excluded from export all tables, types, views, functions and packages + that contains a $ character. Most of the time they don't need to be + exported. + - PLSQL: add automatic conversion of Oracle SYS_REFURSOR as PostgreSQL + REFCURSOR. + - Rewrite entirely the parser of DBMS_OUTPUT du to concatenation errors. + - PLSQL: add automatic replacement of some Oracle exception errors: + INVALID_CURSOR=>INVALID_CURSOR_STATE, ZERO_DIVIDE=>DIVISION_BY_ZERO, + STORAGE_ERROR=>OUT_OF_MEMORY. + +UPGRADE: Ora2Pg.pm and Ora2Pg/PGSQL.pm have changed so they must be overwritten. +There's also changes in the configuration file and documentation has changed as +well. Backward compatibility is fully preserved. + + +2011 07 01 - 8.5 + + - When FILE_PER_FUNCTION is activated and export type is PACKAGE, Ora2Pg + will now save all functions/procedures of a package body into a + directory named as the package name and into different files. This + will allow to load each function separatly or load them all with the + OUTPUT SQL script generated by Ora2Pg. + - Fix Oracle package body parsing failure when a procedure is declared + inside an other. + - Add new configuration options FILE_PER_CONSTRAINT and FILE_PER_INDEX + to generate three files during the schema extraction. One for the + 'CREATE TABLE' statements, one for the constraints (primary keys, + foreign keys, etc.) and the last one for indices. Thanks to Daniel + Scott for the feature request. + - Allow to process PL/SQL Oracle code from file instead of a database + to apply Ora2Pg code conversion. Thank to Mindy Markowitz for the + feature request. See -i or --input_file command line option to ora2pg + perl script or INPUT_FILE new configuration option. + - Add new configuration directive STANDARD_CONFORMING_STRINGS that is + used only during DATA export type to build INSERT statements. This + should avoid 'WARNING: nonstandard use of \\ in a string literal'. + Please check that this behavior is backward compatible with your + PostgreSQL usage as this is enabled by default now. + +UPGRADE: The new features has changed Ora2Pg.pm and ora2pg.pl so that they must +be overwritten. There's also changes in the configuration file and documentation +has changed as well. Take care of backward compatibility with escaped strings in +DATA export type and the new behavior on PACKAGE export. + +2011 06 07 - 8.4 + + - Moves Ora2Pg to SourceForge.net. + - Fix an issue on setting owner in "ALTER SEQUENCE ... SET OWNER TO". + Thanks to Herve Girres for the report. + - Bugfix on lower case convertion for check constraints extraction. + Thanks to Alexander Korotkov for the patch. + +UPGRADE: There's no new functionality, this is a bug fix release. + +2011 05 11 - 8.3 + + - Fix issue on inherited user defined types converted to inherited tables. + Add comment on unsupported inherited type in PostgreSQL too. Thanks to + Mathieu Wingel for the report. + - Fix issue on column default values. Oracle all this kind of strange + syntax: counter NUMBER(4) default '' not null, that was translated to + counter smallint DEFAULT '' by Ora2Pg. Thanks to Mathieu Wingel this is + now rewritten as DEFAULT NOT NULL. + - Fix case sensitivity on create view when there was double quote on the + column name statement part. Thanks to Mathieu Wingel or the report. + - Fix bad patch applied on column name case sensitivity issue during check + constraint export. Thanks to Philippe Rimbault for the report. + - Fix bug on package export introduced into version v8.2. The issue was + related to end of package procedure detection. hanks to Mathieu Wingel + or the report. + +UPGRADE: There's no new functionality, this is a bug fix release and every one +should upgrade to it. + + +2011 05 01 - 8.2 + + - PLSQL: automatic replacement of EXIT WHEN cursor%NOTFOUND; by Pg + synthax: IF NOT FOUND THEN EXIT; END IF;. Works with additional + condition too. + - PLSQL: Automatic replacement of SQL%NOTFOUND by NOT FOUND. + - PLSQL: Add detection of TOO_MANY_ROW to NO_DATA_FOUND to add STRICT. + - Completely rewrite the parsing of Oracle package body to handle all + cases and especially prodedure declared into an other procedure. + Those procedure are renamed INTERNAL_FUNCTION and must be rewritten. + - Fix type usage of ora2pg Perl script. + - Add a new directive FORCE_OWNER. By default the owner of the database + objects is the one you're using to connect to PostgreSQL. If you use + an other user (postgres for exemple) you can force Ora2Pg to set the + object owner to be the one used in the Oracle database by setting the + directive to 1, or to a completely different username by setting the + directive value to that username. Thanks to Herve Girres from Meteo + France for the suggestion and patch. + - Add --forceowner or -f command line option to ora2pg program. + - Add SHOW_ENCODING extract type to return the Oracle session encoding. + For example it can return: NLS_LANG AMERICAN_AMERICA.AL32UTF8 + - Remove SYS_EXTRACT_UTC from index creation as Pg always stores them + in UTC. Thanks to Daniel Scott for the patch. + - In PLSQL code SYS_EXTRACT_UTC is replaced by the Pg syntaxe: + field AT TIME ZONE 'UTC'. + - Fix a pending problem with "Wide character in print at" on COPY mode. + Thanks to Bernd Helmle from Credativ GmbH for the patch. + - PLSQL: Add automatic rewrite of FOR ... IN REVERSE ... into Pg synthax + - Fix column name case sensitivity issue during check constraint export. + Thanks to Daniel Berger for the report. + - Remove the possibility to add comment after a configuration directive + it may not be used and it was generating an issue with the passwords + configuration directives for examples. Thanks to Daniel Berger for the + report. + - Complete rewrite of user defined type extraction. Add support of inherited + type using Oracle UNDER keyword as well as better support to custom type + with BODY. Thanks to Mathieu Wingel for the report. + - Fix case sensitivity on user defined types. Thanks to Mathieu Wingel for + the report. + +UPGRADE: All files have changed so you need a fresh install/upgrade. +Previous release used to remove any string starting from a # in the config file, +this was to allow comments after a configuration directive. This possibility +have been removed in this release so you can no more add comments after a +configuration directive. + + +2011 03 28 - 8.1 + + - Prevent Ora2PG to export twice datas when using FILE_PER_TABLE and + the data output file exist. This is useful in case of export failure + and you don't want to export all data again. This also mean that if + you want to get new data you have to remove the old files before. + - Fix parsing of procedure/function into pl/sql Oracle package. + - Fix bug in IS NULL/IS NOT NULL replacement. Thanks to Jean-Paul Argudo + from Dalibo for the report. + - Add CREATE OR REPLACE on RULE creation. + - Add DROP TRIGGER IF EXISTS before trigger creation. + - Replace Oracle date "0000-00-00" by NULL. + - Fix infinite loop in package/fonction type replacement. + - Add one file per package creation if FILE_PER_FUNCTION is enabled. + - Fix double quote in name of custom type extraction. + - Add extraction of custom type IS VARRAY as an custom type of table + array. Thank to Jean-Paul Argudo from Dalibo for the patch. + - Fix multiple double quote in name of create index definition. + - Apply excluded and limited table to tablespace extraction. + - Fix function and procedure detection/parsing on package content. + - Fix schema prefix in function name declaration in package export. + - PLSQL: Replace some way of extracting date part of a date : + TO_NUMBER(TO_CHAR(...)) rewritten into TO_CHAR(...)::integer when + TO_NUMBER just have one argument. + - Fix Makefile.pl error when trying to modify file ora2pg now renamed + into ora2pg.pl + - Add 3 new export types SHOW_SCHEMA, SHOW_TABLE and SHOW_COLUMN. Those + new extraction keyword are use to only display the requested information + and exit. This allow you to quickly know on what you are going to work. + The SHOW_COLUMN allow a new ora2pg command line option: '--xtable relname' + or '-x relname' to limit the displayed information to the given table. + - Add type replacement for BINARY_INTEGER and PLS_INTEGER as integer. + +UPGRADE: Please make a full upgrade asap to this release. + +2011 03 15 - 8.0 + +This major release simplify and improve Oracle to PostgreSQL export. Ora2Pg v8.x +now assume that you have a modern PostgreSQL release to take full advantage of +the Oracle compatibility effort of the PostgreSQL development team. Ora2Pg since +v8.x release will only be compatible with Pg >= 8.4. + + - Remove addition of AS for alias as with modern PG version this can + be optional (Pg >= 8.4). + - Fix CREATE with missing USER/ROLE for grant extraction. Thanks to + Herve Girres for the report. + - Apply missing psql_pgsql converter to view definition. + - PLSQL : Normalize HAVING ... GROUP BY into GROUP BY ... HAVING clause + - PLSQL : Convert call to Oracle function add_months() in Pg syntax + - PLSQL : Convert call to Oracle function add_years() in Pg syntax + - Apply missing psql_pgsql converter to triggers WHEN clause. + - Fix DECLARE CURSOR rewrite. + - Allow one file per function / procedure / package exported with a new + configuration option FILE_PER_FUNCTION. Useful to editing and testing. + Thank to Jean-Paul Argudo from DALIBO for the feature request. + - The FILE_PER_TABLE configuration option is now also applied to views. + - Remove obsolete PG_SUPPORTS_INOUT as it is supported by with modern + PG version (Pg >= 8.4). + - Remove obsolete PG_SUPPORTS_DEFAULT as it is supported by with modern + PG version (Pg >= 8.4). + - Allow to adjust PostgreSQL client encoding with a new configuration + directive: CLIENT_ENCODING. + - Add TRUNCATE_TABLE configuration directive to add TRUNCATE TABLE + instruction before loading data. + - Add type conversion of Oracle XMLTYPE into PostgreSQL xml type. + - PLSQL: SYSDATE is now replaced by LOCALTIMESTAMP to not use timezone. + Thanks to Jean-Paul Argudo from DALIBO for the report. + - Use 'CREATE OR REPLACE' on create trigger function instruction. + - Fix prefixing by OUTPUT_DIR when file per table/function is enabled. + - Use 'CREATE OR REPLACE' on create view. + - PLSQL_PGSQL is now enabled by default. If you want to export Oracle + original function/procedure/package code, disable it. + - PLSQL: WHERE|AND ROWNUM = N; is automatically replaced by LIMIT N; + - PLSQL: Rewrite comment in CASE between WHEN and THEN that makes Pg + parser unhappy. + - PLSQL: Replace SQLCODE by SQLSTATE + +UPGRADE: You must reinstall all and review your configuration file + +2011 02 14 - 7.3 + + - Remove PG_SUPPORTS_INOUT, now Ora2Pg assumes the PostgreSQL database + destination support it (Pg > v8.1). + - Remove PG_SUPPORT_ROLES, now Ora2Pg assumes the PostgreSQL database + destination support it (Pg > v8.1). + - Complete rewrite of the GRANT (user/role/grant) export type. It now + should be only related to the current Oracle database. Note that do + not try to import rights asis as you may import have errors or worse + miss handling of the rights! Just remember that for example in Oracle + a schema is nothing else than a user so it must not be imported like + this. + - Fix multiple errors in partitionning definition. Thank to Reto Buchli + for the report. + - PLSQL: reordering cursor Oracle declaration "DECLARE CURSOR name" into + "DECLARE name CURSOR". Thank to Reto Buchli (WSL IT) for the report. + - Fix miss handling of DEFAULT parameters value in convert_function(). + Thanks to Leonardo Cezar for the patch. + - Fix Oracle tablespace export where Pg tablespace location was based on + Oracle filename. This fix extract the path and replace the filename + with tablespace name. Thank to Reto Buchli (WSL IT) for the report. + - Fix parsing of ending function code. Thanks to Leonardo Cezar for the + patch. + - Fix call to _convert_procedure() that is in fact the same function as + _convert_function(). Thanks to Leonardo Cezar for the report. + - Fix multiple call on tablespace alter index on the same object. Thank + to Reto Buchli (WSL IT) for the report. + - PSQL: Rewrite RAISE EXCEPTION concatenations. Double pipe (||) are + replaced by % and value is set as parameter a la sprintf. Thank to + Reto Buchli (WSL IT) for the report. + - Add missing comment of PARTITION export type into configutation file. + - Complete rewrite of the table partition export part has it was not + handling all case and was really buggy. + - PLSQL: add normalisation of the to_date() function. + - Ora2Pg now warns during grant export when it is not connected as an + Oracle DBA user. GRANT export need rights of Oracle DBA or it fail. + - Fix install of changelog into Makefile.PL, name was wrong. Thanks to + Julian Moreno Patino for the patch. + + +2011 01 12 - 7.2 + + - Fix escaping of BLOB/RAW to bytea data import causing import to crash. + Thanks to Ozmen Emre Demirkol for the report. + - Add support to default value into functions parameter (PG >= 8.4). + Can be activated with a new configuration directive: PG_SUPPORTS_DEFAULT. + Default is 1, activated. + - Fix bad ending of exported function: remove trailing chars after END. + - Add support to WHEN clause on triggers (PG >= 9.0), can be activated + with a new configuration directive: PG_SUPPORTS_WHEN. + - Add support to INSTEAD OF usage on triggers (incoming PG >= 9.1). Can + be activated with a new configuration directive: PG_SUPPORTS_INSTEADOF. + - Fix error using SKIP directive. Thanks to Laurent Renard from Cap Gemini + for the report. + - Fix missing perl object instance in format_data() function. + - Fix duplicate procedure or function when export type use both FUNCTION + and PROCEDURE. + +2010 12 04 - 7.1 + + - Improve direct DBD::Pg data export/import speed by 10. + - Add --section=3 in pod2man call into Makefile.PL. Thanks to Julian + Moreno Patino for the report. + - Renamed ChangeLog into changelog to avoid upstream warning with Debian + package. Thanks to Julian Moreno Patino for the suggestion. + - Fix some spelling mistakes in doc/Ora2Pg.pod. Thanks to Julian Moreno + Patino for the fix. + - Fix release version into Ora2Pg.pm and PLSQL.pm, was still in 6.5. + - Fix direct data export/import using DBD::Pg. Thanks to Laurent Renard + from Cap Gemini for the report. + - Fix drop/create contraints and index during direct data export/import + using DBD::Pg. Thanks to Thierry Grasland from Cap Gemini for the report. + +2010 11 23 - 7.0 + + - Rename ora2pg perl script into ora2pg.pl in sources because Windows + users can't extract the tarball. During install it is renamed into + ora2pg. Thanks to Andrew Marlow for the report. + - Fix doinst.sh for SlackWare Slackbuid packaging. + - The DEFER_FKEY configuration directive has been fixed as it only + works in a transaction. Note that foreign keys must have been created + as DEFERRABLE or it also will not works. Thanks to Igor Gelman for the + report. + - Add DROP_FKEY configuration directive to force deletion of foreign keys + before the import and recreate them and the end of the import. This may + help if DEFER_FKEY not works for you. + - Add DROP_INDEX configuration directive to force deletion of all indexes + except the automatic index (primary keys) before data import and to + recreate them at end. This can be used to gain speed during import. + - Add TSMSYS, FLOWS_020100 and FLOWS_FILES to the owners exclude list. + This concern the SRS$ table and all tables begining with 'WWV_FLOW_' + - Change the way DATA_LIMIT is working. It must be used now to set the + bulk size of tuples return at once. Default is 10000. + - Improve data export speed by 6! The data export code has been entierly + rewritten and the speed gain is really fun. + - Add OUTPUT_DIR configuration directive to set a base directory where all + dumped files must be written. Default: current directory. + - Change value of default numeric(x) type from float to bigint and change + default numeric(x,y) type to double precision. + - Change conversion type for BFILE from text to bytea. + +2010 09 10 - 6.4 + + - Configuration directives SHOWTABLEID, MIN and MAX are now obsolete + and has been definitively removed. They were never used and add too + much confusion. + - Fix bug in column name replacement where table name was also replaced. + Thank to Jean-Paul Argudo from DALIBO for the report. + - Fix case sensitive errata in PG schema search path. Thank to Jean-Paul + Argudo from DALIBO for the report. + - Remove double \n at end of debug message. + - Fix debug mode not activated if the DEBUG directive is enable and + the -d command line is not present. + - Add unbuffered output for debug message. + +UPGRADE: simply override the Ora2Pg.pm Perl module where it is installed. + + +2010 07 22 - 6.3 + + - Fix Oracle 8i compatibility error during schema extraction complaining + that column CHAR_LENGTH doesn't exist. Thanks to Philippe Rimbault for + the report. Note that the error message is still displayed but tagged + as WARNING only. + - Fix error using the IMPORT option on a read_conf method call. Thanks + to Diogo Biazus for the report. + - Fix export of sequences that does not handle maxvalue well and can be + lower than minvalue. Thanks to Nathalie Doremieux for the report. + +UPGRADE: Just override Ora2Pg.pm + +2010 06 15 - 6.2 + + - Change default transaction isolation level from READ ONLY to + SERIALIZABLE to ensure consistency during data export. Thanks to + Hans-Jurgen Schonig from postgresql-support.de + - Add the TRANSACTION configuration directive to allow overriding of + the isolation level. Value can be readonly, readwrite, committed and + serializable. The last is the default. + +2010 05 07 - 6.1 + + - Fix error on partition export following schema definition. + - Add first support to export Oracle user defined types. + - Add CTXSYS,XDB,WMSYS,SYSMAN,SQLTXPLAIN,MDSYS,EXFSYS,ORDSYS,DMSYS, + OLAPSYS to the sysuser default exclusion list. + - PLSQL.pm: Add automatic translation of Oracle raise_application_error + and dup_val_on_index to PG RAISE EXCEPTION and UNIQUE_VIOLATION. + - Change/fallback to a lower case package name (ora2pg-6.x.tar.gz). + - Change default convert type for 'LONG RAW' to bytea. + - Add PG_SCHEMA configuration directive to defined a coma delimited + list of schema to use in SET search_path PostgreSQL command. + + +2010 02 28 - 6.0 + + - Publish a dedicated site to Ora2Pg at http://ora2pg.darold.net/ + - Add export of Oracle table partitoning. See export type PARTITION. + - Add command line arguments to perl script ora2pg. See --help for a + full listing of these option. The most interesting is --type to change + the export type directly at command execution without needing to edit + the configuration file, --plsql to directly enable PLSQL to PLPSQL + code conversion and --source, --user --password to set Oracle data + source. There's also --namespace to set the Oracle schema. + - Create all file for standard Perl module install. Install is now done + with: perl Makefile.PL && make && make install + - Move Ora2Pg license from Perl Artistics to GPLv3. + - Move PLSQL package as Ora2Pg::PLSQL for standard Perl module install. + - Remove use of Perl module String::Random. + - Rename program ora2pg.pl into ora2pg for standard usage. + - Fix extra double quote on column name of index export. Thanks to + Guillaume Lelarge for the patch. + - Add packaging facilities to build RPM, SlackBuild and Debian packages. + - Fix miss handling of Ora2Pg.pm options at object instance init. + - Configuration file ora2pg.conf is now generated by Makefile.PL + +2009 12 18 - 5.5 + + - Fix CONSTANT declaration in Procedure/Function/Package export. + - Fix length of char and varchar on multibyte like UTF8 encoding. Thanks + to Ali Pouya for the patch. + - Fix view export where alias to column in Oracle not use 'AS' and + PostgreSQL required it. Thanks to Ali Pouya for the report. + - Add type replacement of sql variable in PLSQL code (PLSQL.pm). Thanks + to vijay for the patch. + +2009 07 15 - 5.4 + + - Fix bug introduced in multiple output file feature. This bug force + Ora2pg to crach after the first table export when output is wanted in + a single file. Thanks to Leo Mannhart for the report. + - Fix debug filename output on multiple export file. Thanks to Leo + Mannhart for the report. + +2009 07 07 - version 5.3 + + - Fix wrong escaping of data named as column during view export. Thank + to Andrea Agosti for the patch. + - Allow export of datas into one file per table. See FILE_PER_TABLE + configuration directive. Thanks to Alexandre - Aldeia Digital for the + idea. + +2009 06 19 - version 5.2 + + - Fix order of the column name of the view which was not preserved. Now + ordered by COLUMN_ID. Thank to Andrea Agosti for the report. + - Fix case sensitivity in VIEW extraction. Thank to Andrea Agosti for + the patch. + +2009 03 06 - version 5.1 + + - Fix missing -U username at Pg connection. Thanks to Brendan Richards. + - Fix $ENV{ORACLE_HOME} and $ENV{NLS_LANG} to not being overwritten + by configuration settings if they are already defined in environment. + - Fix typo in ora2pg.pl where keep_pkey_names was replaced by + keep_pkeys_name and so prevent use of KEEP_PKEY_NAMES in configuration. + Thanks to Olivier Mazain for the report. + - Configuration file directives are now case insensitive. + - Force $type parameter given to _sql_type() to be uppercase in that + methode instead of during function call. Thanks to Ali Pouya for the + report. + - Modify ora2pg.pl to remove the obsolete call to export_data(). Use + only export_schema() now. + - Modify ora2pg.pl to simplify it. Reading configuration is now done + internally by Ora2Pg.pm as well as all other initialization process. + You can always overwrite all configuration options into call to new + Ora2Pg(). Now ora2pg.pl can be as simple as: + + use Ora2Pg; + my $schema = new Ora2Pg('config' => "/etc/ora2pg.conf"); + $schema->export_schema(); + exit(0); + + This will be less confusing. You can upgrade Ora2Pg.pm without carring + about that, backward compatibility with previous version is preserved. + - Review entire documentation with the great help of Ali Pouya. + - Add type BOOLEAN converted to boolean. + - PG_SUPPORTS_INOUT is now enabled by default in the configuration file + - SQL and PL/SQL to PLPGSQL converter: + .Replace MINUS call to EXCEPT + .Replace DBMS_OUTPUT.put, DBMS_OUTPUT.put_line, DBMS_OUTPUT.new_line + by the PLPGSQL equivalent: RAISE NOTICE + .Rewrite function/procedure/package convertion functions. + This Oracle SQL converter for function/procedure/package is now only + applied if the configuration directive PLSQL_PGSQL is enable, else + these Oracle code are exported as is. Thanks to Ali Pouya for the help. + + - Reserved call to sql transaction only for DATA export type. Others + export type now use \set ON_ERROR_STOP ON. Thanks to Ali Pouya. + - Fix tablespace creation into schema (missing search_path). Thanks to + Olivier Mazain. + - Fix the type returned by the _sql_type() method in the case of a + numeric with null len and pg_numeric_type is set. Thanks to Ali Pouya. + - Change function body delimiter to $body$ to allow use of $$ into the + body as quote replacement. Thanks to Ali Pouya. + - Fix returns type from function. If multiple OUT parameters: RECORD, + if only one OUT parameter, return his type. If no out parameter: return + VOID. Thanks to Ali Pouya. + - Fix export DATA when the name of a column in the table match COMMENT, + AUDIT or any other defined reserved words. These reserved words are + defined in a new configuration variable ORA_RESERVED_WORDS. It accept + a list of comma separated reserved words. Thanks to Andrea Agosti for + the report. + - Fix configuration parser that omit custom SYSUSERS definition. + +2009 02 13 - version 5.0 + + - Fix places where $self->{prefix} where not used. This prefix is + used to replace DBA_... objects into ALL_... objects. Thanks to Daniel + Scott report and patch. + - Fix some problem on trigger export (missing ending semicolon, return + opaque replaced by return trigger, add missing return new value, single + quote for delimitating the function body hits against quotes inside the + function). Thanks to Luca DallOlio for reports and patches. + - Add first attempt to rewrite plsql code to plpgsql code (see function + plsql_to_plpgsql in new perl module PLSQL.pm). There's a configuration + option named PLSQL_PGSQL to activate the convertion. + +2008 12 16 - version 4.11 + + - Fix Ora2Pg failure on Oracle database with case sensitive tables. + Thanks to Marc Cousin for report and patch. + - Fix missing schema name in query when extract views as tables. + +2008 12 04 - version 4.10 + + - Fix missing replacement of table name on disable triggers when + required. + - Fix some malformed debug output messages. + - Add the capability to extract data from view as if it was a table. + This is usefull if you want to export/import data from an Oracle + view into a Pg table. There's nothing special to do, just to give + the view name into the TABLES configuration directive and set TYPE + to DATA or COPY. If views are not specified in the TABLES directive + there's not view export but only table data. + - Add capability to extract views structure as table schema. There's + nothing special to do, just to give the view name into the TABLES + configuration directive and set TYPE to TABLE. This will not extract + constraints or other table tuning from table used in the view. Thanks + to Groupe SAMSE for the feature request. + +2008 10 27 - version 4.9 + + - Modify the DISABLE_TABLE_TRIGGERS configuration option. Should be now + replaced by DISABLE_TRIGGERS, but compatibility is preserved. + - Add DISABLE_SEQUENCE configuration option to not export alter + sequence after COPY or DATA export. + - Fix extraction of function based index that appears as SYS_NC.... + Thanks to Bozkurt Erkut from SONY for the report + +2008 09 04 - version 4.8 + + - Add SYSUSERS configuration option that allow you to specify a coma + separated list of Oracle System user/schema to exclude from extracted + object. By default it only exclude user SYS,SYSTEM,DBSNMP,OUTLN and + PERFSTAT + - Add support to other binary mode output than ':raw' to avoid the Perl + error message:"Wide character in print". See the BINMODE configuration + directive. This will help a lot if you have UTF-8 records. + Thank to Guillaume Demillecamps for the report. + - Fix double escaping of special character. + Thank to Guillaume Demillecamps for the report. + +2008 01 25 - version 4.7 + + - Add support to regular expressions in the exclusion list. Thanks to + Peter Eisentraut + - Fix misformatted SQL string in function _extract_sequence_info. + Thanks to Bernd Helmle. + - Add escaping of backslash on COPY output. Thanks to Peter Eisentraut + +2008 01 03 - version 4.6 + + - Applied a patch to add ALTER SEQUENCE statements to the dump to + adjust the sequence to the correct values after DATA and COPY dumps. + Thanks to Bernd Helmle. + - Applied a patch which fixes problems with broken COPY output when + extracting data from Orace databases with embedded tabs, carriage + returns and line feeds. Thanks to Bernd Helmle. + - Move the project to PgFoundry + +2007 06 20 - version 4.5 + + - Fix columns order in index extraction. Thanks to Ugo Brunel from BULL. + +2007 04 30 - version 4.4 + + - Fix missing single quote in role extraction. + - Add configuration directive NOESCAPE to enable/disable + escaping characters during data extraction. Default is enabled. + - Add TIMESTAMP, BINARY_FLOAT and BINARY_DOUBLE data type translation. + - Add DATA_TYPE configuration directive to allow user defined data type + translation. + - Add NLS_LANG configuration directive to set Oracle database encoding + and enforce a default language-setting in ora2pg.pl. Thanks to Lars + Weber + +2007 04 03 - version 4.3 + + - Fix duplicate view export. Add schema selector to views. Thank to + Ugo BRUNEL from BULL for the fix. + - Remove 'use strict' to prevent failure on certain condition. + Thank to Andrea Schnabl for the report. + +2006 06 08 - version 4.2 + + - Fix a miss taping on constraint type search that convert unique key + to primary key. Thank to Ugo BRUNEL (BULL) for the patch. + - Fix case sensitivity on CHECK constraint that could cause problem when + check value is uppercase. Thank to Ugo BRUNEL (BULL) for the patch. + +2006 03 28 - version 4.1 + + - Fix a problem when using data_limit and where clause. Thank to + Rene Bentzen for the patch. + - Add enable/disable trigger on data import. Thank to Bernd Helmle. + - Fix escaping of chr(13) MS crashing data import into PG. Thank + to Ugo Brunel (BULL). + +2006 03 22 - version 4.0 + + - Add validation of the requested schema in the database before all. + Thanks to Max Walton for the idea. + - Add multiple export type at the same time. Thanks to Max Walton + for the idea. + - Add support for in/out/inout function parameter. See PG_SUPPORTS_INOUT + configuration option. Thanks to Bernd Helmle for this great + contribution/patch. + - Add support for ROLES with Pg v8.1+. See PG_SUPPORTS_ROLE configure + option. + +2006 02 10 - version 3.4 + + This release add better support to Oracle grant, function and grant + extraction. Great thanks to the Pg team! + + - Add preservation of oracle primary key names. See KEEP_PKEY_NAMES + configuration option. Thanks to Antonios Christofides for this patch. + - Fix bug in case insensitive check constrainte. Thanks to Wojciech + Szenajch for the patch. + - Fix saving data to files correctly (binmod) when the oracle database + contains utf8 chars. Thanks to Richard Chen for the report. + - Fix bug on view extraction when a column contains the word WITH. + Thanks to Richard Chen for the patch. + - Fix wrong mapping between tge data type in Oracle "number(10)" and + Postgresql, which should be "integer" and not "bigint". Thanks to + Sergio Freire for the patch. + - Fix bug in EXCLUDE configuration directive parsing. Thanks to Matt + Miller for the patch. + +2005 02 22 - version 3.3 + + - Fix bug "Modification of a read-only value attempted" + +2005 02 11 - version 3.2 + + - Fix patch error on column position sort + - Replace 'now' by CURRENT_TIMESTAMP on SYSDATE replacement + - Fix bytea type that was not quoted. + +2005 02 10 - version 3.1 + + - Fix bug on deferrable constraint. Thanks to Antonios Christofide for + the patch. + - Fix problem on defer_fkey that should be in a transaction. Thanks to + Antonios Christofide for the patch. + - Add sort by column position during schema extraction. + - Add support to SYSDATE. Thanks to David Cotter-Alatto Technologies Ltd + +2004 12 24 - version 3.0 + + - Add 'TABLESPACE' extraction type to create PostgreSQL v8 tablespace. + +2004 12 24 - version 2.9 + + - Debuging output rewrite. Thanks to Antonios Christofide for help. + - Add 'PG_NUMERIC_TYPE' configuration option to replace portable + numeric type into PostgreSQL internal type (smallint, integer, + bigint, real and float). + +2004 12 24 - version 2.8 + + - Fix/add support to data export of type BLOB, RAW and LONG RAW. + Thanks to Antonios Christofide for help. + +2004 12 23 - version 2.7 + + - Add 'FKEY_DEFERRABLE' configuration option to force foreign key + constraints to be exported as deferrable. Thanks to Antonios + Christofide for help. + - Add 'DEFER_FKEY' configuration option to defer all foreign key + constraints during data export. Thanks to Antonios Christofide + for help. + +2004 12 23 - version 2.6 + + - Fix duplicate output during export. Thanks to Adriano Bonat for the + report. + - Fix data limit infinite loop during data extraction. Thanks to Thomas + REISS for the report. + - Add 'GEN_USER_PWD' configuration option allowing to generate a random + password. Thanks to Antonios Christofide for help. + (Require String::Random from CPAN). + - Fix USER/ROLES/GRANT extraction problem. Now all users are dumped. + All roles are translated to PostgreSQL groups. All grants are + exported. YOU MUST EDIT the output file to rewrite real privilege + and match your needs. Thanks to Antonios Christofide for help. + - Fix split COPY export into multiple transaction for large data export. + The number of row per transaction is set to 'DATA_LIMIT' value. A + value of O mean all in a single transaction. + +2004 10 13 - version 2.5 + + - Fix extraction problem when the connection to Oracle DB is not as DBA. + +2004 08 22 - version 2.4 + + - Fix bug in DBI errstr call. + - Add CASE_SENSITIVE configuration option to allow case sensitivity on + Add a new configuration directive 'USER_GRANTS' to do that. Thanks to + Octavi Fors for the report. + object name. Thanks to Thomas Wegner. + - Fix major bug in unique keys extraction. Thanks to Andreas Haumer and + Marco Lombardo for their great help. + - Add CHECK constraint extration. Thanks again to Andreas Haumer. + - Add IMPORT configuration option to include common configuration file + throught multiple configuration files.Thanks to Adam Sah and Zedo Inc. + - Add SKIP configuration option to turning off extraction of certain + - schema features. Thanks to Adam Sah and Zedo Inc. + - Fix bug in excluded tables + - Fix backslash escaping. Thanks to Adam Sah and Zedo Inc. + - Add REPLACE_TABLES configuration option to change table name during + data extraction. + - Add REPLACE_COLS configuration option to change columns name during + data extraction. + - Add WHERE configuration option to add where clause to each table or + specific tables during extraction. Usefull for replication. Thanks + to Adam Sah and Zedo Inc. + - Add progress indicators (per 1000 rows) and performance results + during data extraction in debug mod. Thanks to Adam Sah and Zedo Inc. + - Add Gzip and Bzip2 compress to output file if extension .gz or .bz2. + Gzip compress require perl module Compress::Zlib from CPAN. Thanks + to Adam Sah for the idea. + +2004 04 13 - Version 2.3 + + - Fix bug in date/time conversion when using data export limit. Thanks + to Andreas Haumer. + - Add sort order when extracting tables and data to respect the TABLES + limited extraction array write order. Usefull if you have foreign key + constraints. Thanks to Andreas Haumer for the idea. + +2004 04 13 - Version 2.2 + + - Add EXCLUDE configuration option to allow table exclusion + from all extraction. + - Fix a bug in escaping single quote on data export. + +2004 03 09 - Version 2.1 + + - Fix COPY output by replacing special character. + - Add configuration file usefull for people who don't have Perl in mind + Thank's to Tanya Krasnokutsky to force me to do that :-) + - Fix other minor problem. + +2002 12 26 - Version 2.0 + + - Clean code. + - Fix COPY output on column value with EOL and add column naming. + - Add support to the PostgreSQL 7.3 schema. So Oracle schema can now be + exported. (see export_schema init option) + - Remove data extraction limit (old default: 10) so each tuple will be + dump by default. + +2002 12 03 - Version 1.12 + + I have fixed 2 bugs when using it against Oracle 817R3 on linux. + + - Fix problem regarding RI constraints, the owner name was not + getting into the sql statement. Thank to Ian Boston. + - Moved all the RI constraints out of the create table statement. + Thank to Ian Boston for this contribution. This was a major request + from Ora2pg users. + +2002 09 27 - Version 1.11 + + - Fix a problem when retrieving package+package body. Thanks to Mike + WILHELM-HILTZ. + - Set LongReadLen to 100000 when exporting table information. Many + users reports this kind of error: A-01406 LongReadLen too small and/or + LongTruncOk not set. This should fix the problem else you must + increase the value. + - Filtering by owner for better performance when retreiving database + schema. Thanks to Jefferson MEDEIROS. + +2002 07 29 - Version 1.10 + + - Fix a problem with local settings regarding decimal separator (all , + are changed to .) Thank to Jan Kester. + +2002 06 04 - Version 1.9 + + - Fix a problem on exporting data which fill NULL instead of 0 or + empty string. Thanks to Jan Kester. + - Add time + date when export data [ tochar('YYYY-MM-DD HH24:MI:SS') ]. + Thanks to Paolo Mattioli. + +2002 03 05 - Version 1.8 + + - Add Oracle type FLOAT conversion to float8. + - Add column alias extraction on view. + Thanks to Jean-Francois RIPOUTEAU + - Add PACKAGE extraction (type => DATA). + +2002 02 14 - Version 1.7 + + - Remove export of OUTLINE object type. Thanks to Jean-Paul ARGUDO. + +2002 01 07 - Version 1.6 + + - Fix problem exporting NULL value. Thanks to Stephane Schildknecht. + +2001 12 28 - Version 1.5 + + - Fix LongReadLen problem when exporting Oracle data on LONG and LOB + types. Thanks to Stephane Schildknecht for report and test. + - Add more precision on NUMBER type conversion + - Add conversion of type LONG, LOB, FILE + - Fix a problem when extracting data, sometime table could need to be + prefixed by the schema name. + - Fix output of Oracle data extraction. It now require a call to + function export_data(). + +2001 06 27 - Version 1.4 + + - Add online Oracle data extraction and insertion into PG database. + - Data export as insert statement (type => DATA) + - Data export as copy from stdin statement (type => COPY) + +2001 06 20 - Version 1.3 + + - Grant/privilege extraction are now done separatly with option + type=>'GRANT' + - Sequence extraction with the option type=>'SEQUENCE' + - Trigger extraction with the option type=>'TRIGGER' + - Function extraction with the option type=>'FUNCTION' and + type=>'PROCEDURE' + - Complete rewrite of the foreign key extraction + - Fix incorrect type translation and many other bug fix + - Add schema only extraction by option schema => 'MYSCHEM' + +2001 05 11 - Version 1.2 + + - Views extraction is now really done with the option type=>'VIEW' + - Add indexes extraction on tables. + - Changes name of constraints, default is now used. + - Add debug printing to see that the process is running :-) + - Add extraction of only required tablename. + - Add extraction of only n to n table indice. Indices of extraction + can be obtained with the option showtableid set to 1. + - Fix print of NOT NULL field. + - Complete rewrite of the grant extraction + - Complete rewrite of most things + +2001 05 09 - Version 1.1 + + - Add table grant extraction based on group. + Oracle ROLES are exported as groups in PG + +2001 05 09 - Initial version 1.0 + +------------------------------------------------------------------------------ + +Special thanks to Ali Pouya for documentation review. All my recognition +to Ali Pouya and Olivier Mazain for their great work in the package and +function export. Thanks to Jean-Paul Argudo for the time spent to heavily +testing Ora2Pg. + +Special thanks to Josian Larcheveque and Stephane Silly as Oracle DBA +and their "patience". + +Special Thanks to Dominique Legendre for his help on Spatial support and +all the tests performed. + +Many thanks for all congratulation message, idea and bug report+fix I received. + +Very special thanks to Jean-Paul Argudo that represent Ora2Pg at Linux Solution Paris 2005. + +Gilles DAROLD + diff --git a/doc/Ora2Pg.pod b/doc/Ora2Pg.pod new file mode 100644 index 0000000000000000000000000000000000000000..cc18f84432ef8489488526bb4e84b2e4cb465144 --- /dev/null +++ b/doc/Ora2Pg.pod @@ -0,0 +1,3122 @@ +=head1 NAME + +Ora2Pg - Oracle to PostgreSQL database schema converter + + +=head1 DESCRIPTION + +Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL +compatible schema. It connects your Oracle database, scans it automatically +and extracts its structure or data, then generates SQL scripts that you can +load into your PostgreSQL database. + +Ora2Pg can be used for anything from reverse engineering Oracle database to +huge enterprise database migration or simply replicating some Oracle data into +a PostgreSQL database. It is really easy to use and doesn't require any Oracle +database knowledge other than providing the parameters needed to connect to the +Oracle database. + + +=head1 FEATURES + +Ora2Pg consist of a Perl script (ora2pg) and a Perl module (Ora2Pg.pm), the +only thing you have to modify is the configuration file ora2pg.conf by setting +the DSN to the Oracle database and optionally the name of a schema. Once that's +done you just have to set the type of export you want: TABLE with constraints, +VIEW, MVIEW, TABLESPACE, SEQUENCE, INDEXES, TRIGGER, GRANT, FUNCTION, PROCEDURE, +PACKAGE, PARTITION, TYPE, INSERT or COPY, FDW, QUERY, KETTLE, SYNONYM. + +By default Ora2Pg exports to a file that you can load into PostgreSQL with the +psql client, but you can also import directly into a PostgreSQL database by +setting its DSN into the configuration file. With all configuration options of +ora2pg.conf you have full control of what should be exported and how. + +Features included: + + - Export full database schema (tables, views, sequences, indexes), with + unique, primary, foreign key and check constraints. + - Export grants/privileges for users and groups. + - Export range/list partitions and sub partitions. + - Export a table selection (by specifying the table names). + - Export Oracle schema to a PostgreSQL 8.4+ schema. + - Export predefined functions, triggers, procedures, packages and + package bodies. + - Export full data or following a WHERE clause. + - Full support of Oracle BLOB object as PG BYTEA. + - Export Oracle views as PG tables. + - Export Oracle user defined types. + - Provide some basic automatic conversion of PLSQL code to PLPGSQL. + - Works on any platform. + - Export Oracle tables as foreign data wrapper tables. + - Export materialized view. + - Show a report of an Oracle database content. + - Migration cost assessment of an Oracle database. + - Migration difficulty level assessment of an Oracle database. + - Migration cost assessment of PL/SQL code from a file. + - Migration cost assessment of Oracle SQL queries stored in a file. + - Generate XML ktr files to be used with Penthalo Data Integrator (Kettle) + - Export Oracle locator and spatial geometries into PostGis. + - Export DBLINK as Oracle FDW. + - Export SYNONYMS as views. + - Export DIRECTORY as external table or directory for external_file extension. + - Full MySQL export just like Oracle database. + - Dispatch a list of SQL orders over multiple PostgreSQL connections + - Perform a diff between Oracle and PostgreSQL database for test purpose. + +Ora2Pg does its best to automatically convert your Oracle database to PostgreSQL +but there's still manual works to do. The Oracle specific PL/SQL code generated +for functions, procedures, packages and triggers has to be reviewed to match +the PostgreSQL syntax. You will find some useful recommendations on porting +Oracle PL/SQL code to PostgreSQL PL/PGSQL at "Converting from other Databases +to PostgreSQL", section: Oracle (http://wiki.postgresql.org/wiki/Main_Page). + +See http://ora2pg.darold.net/report.html for a HTML sample of an Oracle database +migration report. + +=head1 INSTALLATION + +All Perl modules can always be found at CPAN (http://search.cpan.org/). Just +type the full name of the module (ex: DBD::Oracle) into the search input box, +it will brings you the page for download. + +Releases of Ora2Pg stay at SF.net (https://sourceforge.net/projects/ora2pg/). + +Under Windows you should install Strawberry Perl (http://strawberryperl.com/) +and the OSes corresponding Oracle clients. Since version 5.32 this Perl +distribution include pre-compiled driver of DBD::Oracle and DBD::Pg. + +=head2 Requirement + +The Oracle Instant Client or a full Oracle installation must be installed on +the system. You can download the RPM from Oracle download center: + + rpm -ivh oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm + rpm -ivh oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm + rpm -ivh oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm + rpm -ivh oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm + +or simply download the corresponding ZIP archives from Oracle download center +and install them where you want, for example: /opt/oracle/instantclient_12_2/ + +You also need a modern Perl distribution (perl 5.10 and more). To connect to a +database and proceed to his migration you need the DBI Perl module > 1.614. +To migrate an Oracle database you need the DBD::Oracle Perl modules to be +installed. To migrate a MySQL database you need the DBD::MySQL Perl modules. +These modules are used to connect to the database but they are not mandatory +if you want to migrate DDL input files. + +To install DBD::Oracle and have it working you need to have the Oracle client +libraries installed and the ORACLE_HOME environment variable must be defined. + +If you plan to export a MySQL database you need to install the Perl module +DBD::mysql which requires that the mysql client libraries are installed. + +On some Perl distribution you may need to install the Time::HiRes Perl module. + +If your distribution doesn't include these Perl modules you can install them +using CPAN: + + perl -MCPAN -e 'install DBD::Oracle' + perl -MCPAN -e 'install DBD::MySQL' + perl -MCPAN -e 'install Time::HiRes' + +otherwise use the packages provided by your distribution. + +=head2 Optional + +By default Ora2Pg dumps export to flat files, to load them into your PostgreSQL +database you need the PostgreSQL client (psql). If you don't have it on the +host running Ora2Pg you can always transfer these files to a host with the psql +client installed. If you prefer to load export 'on the fly', the perl module +DBD::Pg is required. + +Ora2Pg allows you to dump all output in a compressed gzip file, to do that you +need the Compress::Zlib Perl module or if you prefer using bzip2 compression, +the program bzip2 must be available in your PATH. + +If your distribution doesn't include these Perl modules you can install them +using CPAN: + + perl -MCPAN -e 'install DBD::Pg' + perl -MCPAN -e 'install Compress::Zlib' + +otherwise use the packages provided by your distribution. + +=head2 Installing Ora2Pg + +Like any other Perl Module Ora2Pg can be installed with the following commands: + + tar xjf ora2pg-x.x.tar.bz2 + cd ora2pg-x.x/ + perl Makefile.PL + make && make install + +This will install Ora2Pg.pm into your site Perl repository, ora2pg into +/usr/local/bin/ and ora2pg.conf into /etc/ora2pg/. + +On Windows(tm) OSes you may use instead: + + perl Makefile.PL + dmake && dmake install + +This will install scripts and libraries into your Perl site installation +directory and the ora2pg.conf file as well as all documentation files +into C:\ora2pg\ + +To install ora2pg in a different directory than the default one, simply +use this command: + + perl Makefile.PL PREFIX= + make && make install + +then set PERL5LIB to the path to your installation directory before using +Ora2Pg. + + export PERL5LIB= + ora2pg -c config/ora2pg.conf -t TABLE -b outdir/ + +=head2 Packaging + +If you want to build the binary package for your preferred Linux distribution +take a look at the packaging/ directory of the source tarball. There is +everything to build RPM, Slackware and Debian packages. See README file in +that directory. + +=head2 Installing DBD::Oracle + +Ora2Pg needs the Perl module DBD::Oracle for connectivity to an Oracle database +from perl DBI. To get DBD::Oracle get it from CPAN a perl module repository. + +After setting ORACLE_HOME and LD_LIBRARY_PATH environment variables as root +user, install DBD::Oracle. Proceed as follow: + + export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib + export ORACLE_HOME=/usr/lib/oracle/12.2/client64 + perl -MCPAN -e 'install DBD::Oracle' + +If you are running for the first time it will ask many questions; you can keep +defaults by pressing ENTER key, but you need to give one appropriate mirror +site for CPAN to download the modules. Install through CPAN manually if the +above doesn't work: + + #perl -MCPAN -e shell + cpan> get DBD::Oracle + cpan> quit + cd ~/.cpan/build/DBD-Oracle* + export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib + export ORACLE_HOME=/usr/lib/oracle/11.2/client64 + perl Makefile.PL + make + make install + +Installing DBD::Oracle require that the three Oracle packages: instant-client, +SDK and SQLplus are installed as well as the libaio1 library. + +If you are using Instant Client from ZIP archives, the LD_LIBRARY_PATH and +ORACLE_HOME will be the same and must be set to the directory where you have +installed the files. For example: /opt/oracle/instantclient_12_2/ + +=head1 CONFIGURATION + +Ora2Pg configuration can be as simple as choosing the Oracle database to export +and choose the export type. This can be done in a minute. + +By reading this documentation you will also be able to: + + - Select only certain tables and/or column for export. + - Rename some tables and/or column during export. + - Select data to export following a WHERE clause per table. + - Delay database constraints during data loading. + - Compress exported data to save disk space. + - and much more. + +The full control of the Oracle database migration is taken though a single +configuration file named ora2pg.conf. The format of this file consist in a +directive name in upper case followed by tab character and a value. +Comments are lines beginning with a #. + +There's no specific order to place the configuration directives, they are +set at the time they are read in the configuration file. + +For configuration directives that just take a single value, you can use them +multiple time in the configuration file but only the last occurrence found +in the file will be used. For configuration directives that allow a list +of value, you can use it multiple time, the values will be appended to the +list. If you use the IMPORT directive to load a custom configuration file, +directives defined in this file will be stores from the place the IMPORT +directive is found, so it is better to put it at the end of the configuration +file. + +Values set in command line options will override values from the configuration +file. + +=head2 Ora2Pg usage + +First of all be sure that libraries and binaries path include the Oracle +Instant Client installation: + + export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib + export PATH="/usr/lib/oracle/11.2/client64/bin:$PATH" + +By default Ora2Pg will look for /etc/ora2pg/ora2pg.conf configuration file, if +the file exist you can simply execute: + + /usr/local/bin/ora2pg + +or under Windows(tm) run ora2pg.bat file, located in your perl bin directory. +Windows(tm) users may also find a template configuration file in C:\ora2pg + +If you want to call another configuration file, just give the path as command +line argument: + + /usr/local/bin/ora2pg -c /etc/ora2pg/new_ora2pg.conf + +Here are all command line parameters available when using ora2pg: + +Usage: ora2pg [-dhpqv --estimate_cost --dump_as_html] [--option value] + + -a | --allow str : Comma separated list of objects to allow from export. + Can be used with SHOW_COLUMN too. + -b | --basedir dir: Set the default output directory, where files + resulting from exports will be stored. + -c | --conf file : Set an alternate configuration file other than the + default /etc/ora2pg/ora2pg.conf. + -d | --debug : Enable verbose output. + -D | --data_type STR : Allow custom type replacement at command line. + -e | --exclude str: Comma separated list of objects to exclude from export. + Can be used with SHOW_COLUMN too. + -h | --help : Print this short help. + -g | --grant_object type : Extract privilege from the given object type. + See possible values with GRANT_OBJECT configuration. + -i | --input file : File containing Oracle PL/SQL code to convert with + no Oracle database connection initiated. + -j | --jobs num : Number of parallel process to send data to PostgreSQL. + -J | --copies num : Number of parallel connections to extract data from Oracle. + -l | --log file : Set a log file. Default is stdout. + -L | --limit num : Number of tuples extracted from Oracle and stored in + memory before writing, default: 10000. + -m | --mysql : Export a MySQL database instead of an Oracle schema. + -n | --namespace schema : Set the Oracle schema to extract from. + -N | --pg_schema schema : Set PostgreSQL's search_path. + -o | --out file : Set the path to the output file where SQL will + be written. Default: output.sql in running directory. + -p | --plsql : Enable PLSQL to PLPGSQL code conversion. + -P | --parallel num: Number of parallel tables to extract at the same time. + -q | --quiet : Disable progress bar. + -r | --relative : use \ir instead of \i in the psql scripts generated. + -s | --source DSN : Allow to set the Oracle DBI datasource. + -t | --type export: Set the export type. It will override the one + given in the configuration file (TYPE). + -T | --temp_dir DIR: Set a distinct temporary directory when two + or more ora2pg are run in parallel. + -u | --user name : Set the Oracle database connection user. + ORA2PG_USER environment variable can be used instead. + -v | --version : Show Ora2Pg Version and exit. + -w | --password pwd : Set the password of the Oracle database user. + ORA2PG_PASSWD environment variable can be used instead. + --forceowner : Force ora2pg to set tables and sequences owner like in + Oracle database. If the value is set to a username this one + will be used as the objects owner. By default it's the user + used to connect to the Pg database that will be the owner. + --nls_lang code: Set the Oracle NLS_LANG client encoding. + --client_encoding code: Set the PostgreSQL client encoding. + --view_as_table str: Comma separated list of views to export as table. + --estimate_cost : Activate the migration cost evaluation with SHOW_REPORT + --cost_unit_value minutes: Number of minutes for a cost evaluation unit. + default: 5 minutes, corresponds to a migration conducted by a + PostgreSQL expert. Set it to 10 if this is your first migration. + --dump_as_html : Force ora2pg to dump report in HTML, used only with + SHOW_REPORT. Default is to dump report as simple text. + --dump_as_csv : As above but force ora2pg to dump report in CSV. + --dump_as_sheet : Report migration assessment with one CSV line per database. + --init_project NAME: Initialise a typical ora2pg project tree. Top directory + will be created under project base dir. + --project_base DIR : Define the base dir for ora2pg project trees. Default + is current directory. + --print_header : Used with --dump_as_sheet to print the CSV header + especially for the first run of ora2pg. + --human_days_limit num : Set the number of human-days limit where the migration + assessment level switch from B to C. Default is set to + 5 human-days. + --audit_user LIST : Comma separated list of usernames to filter queries in + the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT + and QUERY export type. + --pg_dsn DSN : Set the datasource to PostgreSQL for direct import. + --pg_user name : Set the PostgreSQL user to use. + --pg_pwd password : Set the PostgreSQL password to use. + --count_rows : Force ora2pg to perform a real row count in TEST action. + --no_header : Do not append Ora2Pg header to output file + --oracle_speed : Use to know at which speed Oracle is able to send + data. No data will be processed or written. + --ora2pg_speed : Use to know at which speed Ora2Pg is able to send + transformed data. Nothing will be written. + +See full documentation at http://ora2pg.darold.net/ for more help or see +manpage with 'man ora2pg'. + +ora2pg will return 0 on success, 1 on error. It will return 2 when a child +process has been interrupted and you've gotten the warning message: + "WARNING: an error occurs during data export. Please check what's happen." +Most of the time this is an OOM issue, first try reducing DATA_LIMIT value. + +For developers, it is possible to add your own custom option(s) in the Perl +script ora2pg as any configuration directive from ora2pg.conf can be passed +in lower case to the new Ora2Pg object instance. See ora2pg code on how to +add your own option. + +Note that performance might be improved by updating stats on oracle: + + BEGIN + DBMS_STATS.GATHER_SCHEMA_STATS + DBMS_STATS.GATHER_DATABASE_STATS + DBMS_STATS.GATHER_DICTIONARY_STATS + END; + +=head2 Generate a migration template + +The two options --project_base and --init_project when used indicate to ora2pg +that he has to create a project template with a work tree, a configuration +file and a script to export all objects from the Oracle database. Here a sample +of the command usage: + + ora2pg --project_base /app/migration/ --init_project test_project + Creating project test_project. + /app/migration/test_project/ + schema/ + dblinks/ + directories/ + functions/ + grants/ + mviews/ + packages/ + partitions/ + procedures/ + sequences/ + synonyms/ + tables/ + tablespaces/ + triggers/ + types/ + views/ + sources/ + functions/ + mviews/ + packages/ + partitions/ + procedures/ + triggers/ + types/ + views/ + data/ + config/ + reports/ + + Generating generic configuration file + Creating script export_schema.sh to automate all exports. + Creating script import_all.sh to automate all imports. + +It create a generic config file where you just have to define the Oracle +database connection and a shell script called export_schema.sh. The sources/ +directory will contains the Oracle code, the schema/ will contains the code +ported to PostgreSQL. The reports/ directory will contains the html reports +with the migration cost assessment. + +If you want to use your own default config file, use the -c option to give +the path to that file. Rename it with .dist suffix if you want ora2pg to +apply the generic configuration values otherwise, the configuration file +will be copied untouched. + +Once you have set the connection to the Oracle Database you can execute the +script export_schema.sh that will export all object type from your Oracle +database and output DDL files into the schema's subdirectories. At end of the +export it will give you the command to export data later when the import of +the schema will be done and verified. + +You can choose to load the DDL files generated manually or use the second +script import_all.sh to import those file interactively. If this kind of +migration is not something current for you it's recommended you to use those +scripts. + + +=head2 Oracle database connection + +There's 5 configuration directives to control the access to the Oracle database. + +=over 4 + +=item ORACLE_HOME + +Used to set ORACLE_HOME environment variable to the Oracle libraries required +by the DBD::Oracle Perl module. + +=item ORACLE_DSN + +This directive is used to set the data source name in the form standard DBI DSN. +For example: + + dbi:Oracle:host=oradb_host.myhost.com;sid=DB_SID;port=1521 + +or + + dbi:Oracle:DB_SID + +On 18c this could be for example: + + dbi:Oracle:host=192.168.1.29;service_name=pdb1;port=1521 + +for the second notation the SID should be declared in the well known file +$ORACLE_HOME/network/admin/tnsnames.ora or in the path given to the TNS_ADMIN +environment variable. + +For MySQL the DSN will lool like this: + + dbi:mysql:host=192.168.1.10;database=sakila;port=3306 + +the 'sid' part is replaced by 'database'. + +=item ORACLE_USER et ORACLE_PWD + +These two directives are used to define the user and password for the Oracle +database connection. Note that if you can it is better to login as Oracle super +admin to avoid grants problem during the database scan and be sure that nothing +is missing. + +If you do not supply a credential with ORACLE_PWD and you have installed the +Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If +ORACLE_USER is not set it will be asked interactively too. + +To connect to a local ORACLE instance with connections "as sysdba" you have to +set ORACLE_USER to "/" and an empty password. + +=item USER_GRANTS + +Set this directive to 1 if you connect the Oracle database as simple user and +do not have enough grants to extract things from the DBA_... tables. It will +use tables ALL_... instead. + +Warning: if you use export type GRANT, you must set this configuration option +to 0 or it will not work. + +=item TRANSACTION + +This directive may be used if you want to change the default isolation level of +the data export transaction. Default is now to set the level to a serializable +transaction to ensure data consistency. The allowed values for this directive +are: + + readonly: 'SET TRANSACTION READ ONLY', + readwrite: 'SET TRANSACTION READ WRITE', + serializable: 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' + committed: 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED', + + +Releases before 6.2 used to set the isolation level to READ ONLY transaction +but in some case this was breaking data consistency so now default is set to +SERIALIZABLE. + +=item INPUT_FILE + +This directive did not control the Oracle database connection or unless it +purely disables the use of any Oracle database by accepting a file as argument. +Set this directive to a file containing PL/SQL Oracle Code like function, +procedure or full package body to prevent Ora2Pg from connecting to an +Oracle database and just apply his conversion tool to the content of the +file. This can be used with the most of export types: TABLE, TRIGGER, PROCEDURE, +VIEW, FUNCTION or PACKAGE, etc. + +=item ORA_INITIAL_COMMAND + +This directive can be used to send an initial command to Oracle, just after +the connection. For example to unlock a policy before reading objects or +to set some session parameters. This directive can be used multiple times. + +=back + +=head2 Data encryption with Oracle server + +If your Oracle Client config file already includes the encryption method, +then DBD:Oracle uses those settings to encrypt the connection while you +extract the data. For example if you have configured the Oracle Client +config file (sqlnet.or or .sqlnet) with the following information: + + # Configure encryption of connections to Oracle + SQLNET.ENCRYPTION_CLIENT = required + SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256, RC4_256) + SQLNET.CRYPTO_SEED = 'should be 10-70 random characters' + +Any tool that uses the Oracle client to talk to the database will be +encrypted if you setup session encryption like above. + +For example, Perl's DBI uses DBD-Oracle, which uses the Oracle client +for actually handling database communication. If the installation of +Oracle client used by Perl is setup to request encrypted connections, +then your Perl connection to an Oracle database will also be encrypted. + +Full details at https://kb.berkeley.edu/jivekb/entry.jspa?externalID=1005 + +=head2 Testing connection + +Once you have set the Oracle database DSN you can execute ora2pg to see if +it works: + + ora2pg -t SHOW_VERSION -c config/ora2pg.conf + +will show the Oracle database server version. Take some time here to test your +installation as most problems take place here, the other configuration +steps are more technical. + +=head2 Troubleshooting + +If the output.sql file has not exported anything other than the Pg transaction +header and footer there's two possible reasons. The perl script ora2pg dump +an ORA-XXX error, that mean that your DSN or login information are wrong, check +the error and your settings and try again. The perl script says nothing and the +output file is empty: the user lacks permission to extract something from +the database. Try to connect to Oracle as super user or take a look at directive +USER_GRANTS above and at next section, especially the SCHEMA directive. + +=over 4 + +=item LOGFILE + +By default all messages are sent to the standard output. If you give a file +path to that directive, all output will be appended to this file. + +=back + +=head2 Oracle schema to export + +The Oracle database export can be limited to a specific Schema or Namespace, +this can be mandatory following the database connection user. + +=over 4 + +=item SCHEMA + +This directive is used to set the schema name to use during export. +For example: + + SCHEMA APPS + +will extract objects associated to the APPS schema. + +When no schema name is provided and EXPORT_SCHEMA is enabled, Ora2Pg +will export all objects from all schema of the Oracle instance with +their names prefixed with the schema name. + +=item EXPORT_SCHEMA + +By default the Oracle schema is not exported into the PostgreSQL database and +all objects are created under the default Pg namespace. If you want to also +export this schema and create all objects under this namespace, set the +EXPORT_SCHEMA directive to 1. This will set the schema search_path at top of +export SQL file to the schema name set in the SCHEMA directive with the default +pg_catalog schema. If you want to change this path, use the directive PG_SCHEMA. + +=item CREATE_SCHEMA + +Enable/disable the CREATE SCHEMA SQL order at starting of the output file. +It is enable by default and concern on TABLE export type. + +=item COMPILE_SCHEMA + +By default Ora2Pg will only export valid PL/SQL code. You can force Oracle to +compile again the invalidated code to get a chance to have it obtain the valid +status and then be able to export it. + +Enable this directive to force Oracle to compile schema before exporting code. +When this directive is enabled and SCHEMA is set to a specific schema name, +only invalid objects in this schema will be recompiled. If SCHEMA is not set +then all schema will be recompiled. To force recompile invalid object in a +specific schema, set COMPILE_SCHEMA to the schema name you want to recompile. + +This will ask to Oracle to validate the PL/SQL that could have been invalidate +after a export/import for example. The 'VALID' or 'INVALID' status applies to +functions, procedures, packages and user defined types. + +=item EXPORT_INVALID + +If the above configuration directive is not enough to validate your PL/SQL code +enable this configuration directive to allow export of all PL/SQL code even if +it is marked as invalid. The 'VALID' or 'INVALID' status applies to functions, +procedures, packages and user defined types. + +=item PG_SCHEMA + +Allow you to defined/force the PostgreSQL schema to use. By default if you set +EXPORT_SCHEMA to 1 the PostgreSQL search_path will be set to the schema name +exported set as value of the SCHEMA directive. + +The value can be a comma delimited list of schema name but not when using TABLE +export type because in this case it will generate the CREATE SCHEMA statement +and it doesn't support multiple schema name. For example, if you set PG_SCHEMA +to something like "user_schema, public", the search path will be set like this: + + SET search_path = user_schema, public; + +forcing the use of an other schema (here user_schema) than the one from Oracle +schema set in the SCHEMA directive. + +You can also set the default search_path for the PostgreSQL user you are using +to connect to the destination database by using: + + ALTER ROLE username SET search_path TO user_schema, public; + +in this case you don't have to set PG_SCHEMA. + +=item SYSUSERS + +Without explicit schema, Ora2Pg will export all objects that not belongs to +system schema or role: + + SYSTEM,CTXSYS,DBSNMP,EXFSYS,LBACSYS,MDSYS,MGMT_VIEW, + OLAPSYS,ORDDATA,OWBSYS,ORDPLUGINS,ORDSYS,OUTLN, + SI_INFORMTN_SCHEMA,SYS,SYSMAN,WK_TEST,WKSYS,WKPROXY, + WMSYS,XDB,APEX_PUBLIC_USER,DIP,FLOWS_020100,FLOWS_030000, + FLOWS_040100,FLOWS_010600,FLOWS_FILES,MDDATA,ORACLE_OCM, + SPATIAL_CSW_ADMIN_USR,SPATIAL_WFS_ADMIN_USR,XS$NULL,PERFSTAT, + SQLTXPLAIN,DMSYS,TSMSYS,WKSYS,APEX_040000,APEX_040200, + DVSYS,OJVMSYS,GSMADMIN_INTERNAL,APPQOSSYS,DVSYS,DVF, + AUDSYS,APEX_030200,MGMT_VIEW,ODM,ODM_MTR,TRACESRV,MTMSYS, + OWBSYS_AUDIT,WEBSYS,WK_PROXY,OSE$HTTP$ADMIN, + AURORA$JIS$UTILITY$,AURORA$ORB$UNAUTHENTICATED, + DBMS_PRIVILEGE_CAPTURE,CSMIG,MGDSYS,SDE,DBSFWUSER + +Following your Oracle installation you may have several other system role +defined. To append these users to the schema exclusion list, just set the +SYSUSERS configuration directive to a comma-separated list of system user to +exclude. For example: + + SYSUSERS INTERNAL,SYSDBA,BI,HR,IX,OE,PM,SH + +will add users INTERNAL and SYSDBA to the schema exclusion list. + +=item FORCE_OWNER + +By default the owner of the database objects is the one you're using to connect +to PostgreSQL using the psql command. If you use an other user (postgres for example) +you can force Ora2Pg to set the object owner to be the one used in the Oracle database +by setting the directive to 1, or to a completely different username by setting the +directive value to that username. + +=item FORCE_SECURITY_INVOKER + +Ora2Pg use the function's security privileges set in Oracle and it is often +defined as SECURITY DEFINER. If you want to override those security privileges +for all functions and use SECURITY DEFINER instead, enable this directive. + +=item USE_TABLESPACE + +When enabled this directive force ora2pg to export all tables, indexes constraint and +indexes using the tablespace name defined in Oracle database. This works only with +tablespace that are not TEMP, USERS and SYSTEM. + +=item WITH_OID + +Activating this directive will force Ora2Pg to add WITH (OIDS) when creating +tables or views as tables. Default is same as PostgreSQL, disabled. + +=item LOOK_FORWARD_FUNCTION + +List of schema to get functions/procedures meta information that are used +in the current schema export. When replacing call to function with OUT +parameters, if a function is declared in an other package then the function +call rewriting can not be done because Ora2Pg only knows about functions +declared in the current schema. By setting a comma separated list of schema +as value of this directive, Ora2Pg will look forward in these packages for +all functions/procedures/packages declaration before proceeding to current +schema export. + +=item NO_FUNCTION_METADATA + +Force Ora2Pg to not look for function declaration. Note that this will prevent +Ora2Pg to rewrite function replacement call if needed. Do not enable it unless +looking forward at function breaks other export. + +=back + +=head2 Export type + +The export action is perform following a single configuration directive 'TYPE', +some other add more control on what should be really exported. + +=over 4 + +=item TYPE + +Here are the different values of the TYPE directive, default is TABLE: + + - TABLE: Extract all tables with indexes, primary keys, unique keys, + foreign keys and check constraints. + - VIEW: Extract only views. + - GRANT: Extract roles converted to Pg groups, users and grants on all + objects. + - SEQUENCE: Extract all sequence and their last position. + - TABLESPACE: Extract storage spaces for tables and indexes (Pg >= v8). + - TRIGGER: Extract triggers defined following actions. + - FUNCTION: Extract functions. + - PROCEDURE: Extract procedures. + - PACKAGE: Extract packages and package bodies. + - INSERT: Extract data as INSERT statement. + - COPY: Extract data as COPY statement. + - PARTITION: Extract range and list Oracle partitions with subpartitions. + - TYPE: Extract user defined Oracle type. + - FDW: Export Oracle tables as foreign table for oracle_fdw. + - MVIEW: Export materialized view. + - QUERY: Try to automatically convert Oracle SQL queries. + - KETTLE: Generate XML ktr template files to be used by Kettle. + - DBLINK: Generate oracle foreign data wrapper server to use as dblink. + - SYNONYM: Export Oracle's synonyms as views on other schema's objects. + - DIRECTORY: Export Oracle's directories as external_file extension objects. + - LOAD: Dispatch a list of queries over multiple PostgreSQl connections. + - TEST: perform a diff between Oracle and PostgreSQL database. + - TEST_VIEW: perform a count on both side of rows returned by views + + +Only one type of export can be perform at the same time so the TYPE directive +must be unique. If you have more than one only the last found in the file will +be registered. + +Some export type can not or should not be load directly into the PostgreSQL +database and still require little manual editing. This is the case for GRANT, +TABLESPACE, TRIGGER, FUNCTION, PROCEDURE, TYPE, QUERY and PACKAGE export types +especially if you have PLSQL code or Oracle specific SQL in it. + +For TABLESPACE you must ensure that file path exist on the system and for +SYNONYM you may ensure that the object's owners and schemas correspond to +the new PostgreSQL database design. + +Note that you can chained multiple export by giving to the TYPE directive a +comma-separated list of export type, but in this case you must not use COPY +or INSERT with other export type. + +Ora2Pg will convert Oracle partition using table inheritance, trigger and +functions. See document at Pg site: +http://www.postgresql.org/docs/current/interactive/ddl-partitioning.html + +The TYPE export allow export of user defined Oracle type. If you don't use the +--plsql command line parameter it simply dump Oracle user type asis else Ora2Pg +will try to convert it to PostgreSQL syntax. + +The KETTLE export type requires that the Oracle and PostgreSQL DNS are defined. + +Since Ora2Pg v8.1 there's three new export types: + + SHOW_VERSION : display Oracle version + SHOW_SCHEMA : display the list of schema available in the database. + SHOW_TABLE : display the list of tables available. + SHOW_COLUMN : display the list of tables columns available and the + Ora2PG conversion type from Oracle to PostgreSQL that will be + applied. It will also warn you if there's PostgreSQL reserved + words in Oracle object names. + +Here is an example of the SHOW_COLUMN output: + + [2] TABLE CURRENT_SCHEMA (1 rows) (Warning: 'CURRENT_SCHEMA' is a reserved word in PostgreSQL) + CONSTRAINT : NUMBER(22) => bigint (Warning: 'CONSTRAINT' is a reserved word in PostgreSQL) + FREEZE : VARCHAR2(25) => varchar(25) (Warning: 'FREEZE' is a reserved word in PostgreSQL) + ... + [6] TABLE LOCATIONS (23 rows) + LOCATION_ID : NUMBER(4) => smallint + STREET_ADDRESS : VARCHAR2(40) => varchar(40) + POSTAL_CODE : VARCHAR2(12) => varchar(12) + CITY : VARCHAR2(30) => varchar(30) + STATE_PROVINCE : VARCHAR2(25) => varchar(25) + COUNTRY_ID : CHAR(2) => char(2) + +Those extraction keywords are use to only display the requested information and +exit. This allows you to quickly know on what you are going to work. + +The SHOW_COLUMN allow an other ora2pg command line option: '--allow relname' +or '-a relname' to limit the displayed information to the given table. + +The SHOW_ENCODING export type will display the NLS_LANG and CLIENT_ENCODING +values that Ora2Pg will used and the real encoding of the Oracle database with +the corresponding client encoding that could be used with PostgreSQL + +Since release v8.12, Ora2Pg allow you to export your Oracle Table definition to +be use with the oracle_fdw foreign data wrapper. By using type FDW your Oracle +tables will be exported as follow: + + CREATE FOREIGN TABLE oratab ( + id integer NOT NULL, + text character varying(30), + floating double precision NOT NULL + ) SERVER oradb OPTIONS (table 'ORATAB'); + +Now you can use the table like a regular PostgreSQL table. + +See http://pgxn.org/dist/oracle_fdw/ for more information on this foreign data +wrapper. + +Release 10 adds a new export type destined to evaluate the content of the +database to migrate, in terms of objects and cost to end the migration: + + SHOW_REPORT : show a detailed report of the Oracle database content. + +Here is a sample of report: http://ora2pg.darold.net/report.html + +There also a more advanced report with migration cost. See the dedicated chapter +about Migration Cost Evaluation. + +=item ESTIMATE_COST + +Activate the migration cost evaluation. Must only be used with SHOW_REPORT, +FUNCTION, PROCEDURE, PACKAGE and QUERY export type. Default is disabled. +You may want to use the --estimate_cost command line option instead to activate +this functionality. Note that enabling this directive will force PLSQL_PGSQL +activation. + +=item COST_UNIT_VALUE + +Set the value in minutes of the migration cost evaluation unit. Default +is five minutes per unit. See --cost_unit_value to change the unit value +at command line. + +=item DUMP_AS_HTML + +By default when using SHOW_REPORT the migration report is generated as simple +text, enabling this directive will force ora2pg to create a report in HTML +format. + +See http://ora2pg.darold.net/report.html for a sample report. + +=item HUMAN_DAYS_LIMIT + +Use this directive to redefined the number of human-days limit where the +migration assessment level must switch from B to C. Default is set to 10 +human-days. + +=item JOBS + +This configuration directive adds multiprocess support to COPY, FUNCTION +and PROCEDURE export type, the value is the number of process to use. +Default is multiprocess disable. + +This directive is used to set the number of cores to used to parallelize +data import into PostgreSQL. During FUNCTION or PROCEDURE export type each +function will be translated to plpgsql using a new process, the performances +gain can be very important when you have tons of function to convert. + +There's no limitation in parallel processing than the number of cores +and the PostgreSQL I/O performance capabilities. + +Doesn't work under Windows Operating System, it is simply disabled. + +=item ORACLE_COPIES + +This configuration directive adds multiprocess support to extract data +from Oracle. The value is the number of process to use to parallelize +the select query. Default is parallel query disable. + +The parallelism is built on splitting the query following of the number +of cores given as value to ORACLE_COPIES as follow: + + SELECT * FROM MYTABLE WHERE ABS(MOD(COLUMN, ORACLE_COPIES)) = CUR_PROC + +where COLUMN is a technical key like a primary or unique key where split +will be based and the current core used by the query (CUR_PROC). + +Doesn't work under Windows Operating System, it is simply disabled. + +=item DEFINED_PK + +This directive is used to defined the technical key to used to split +the query between number of cores set with the ORACLE_COPIES variable. +For example: + + DEFINED_PK EMPLOYEES:employee_id + +The parallel query that will be used supposing that -J or ORACLE_COPIES +is set to 8: + + SELECT * FROM EMPLOYEES WHERE ABS(MOD(employee_id, 8)) = N + +where N is the current process forked starting from 0. + +=item PARALLEL_TABLES + +This directive is used to defined the number of tables that will be processed +in parallel for data extraction. The limit is the number of cores on your machine. +Ora2Pg will open one database connection for each parallel table extraction. +This directive, when upper than 1, will invalidate ORACLE_COPIES but not JOBS, +so the real number of process that will be used is PARALLEL_TABLES * JOBS. + +Note that this directive when set upper that 1 will also automatically enable +the FILE_PER_TABLE directive if your are exporting to files. + +=item DEFAULT_PARALLELISM_DEGREE + +You can force Ora2Pg to use /*+ PARALLEL(tbname, degree) */ hint in each +query used to export data from Oracle by setting a value upper than 1 to +this directive. A value of 0 or 1 disable the use of parallel hint. +Default is disabled. + +=item FDW_SERVER + +This directive is used to set the name of the foreign data server that is used +in the "CREATE SERVER name FOREIGN DATA WRAPPER oracle_fdw ..." command. This +name will then be used in the "CREATE FOREIGN TABLE ..." SQL command. Default +is arbitrary set to orcl. This only concern export type FDW. + +=item EXTERNAL_TO_FDW + +This directive, enabled by default, allow to export Oracle's External Tables as +file_fdw foreign tables. To not export these tables at all, set the directive +to 0. + +=item INTERNAL_DATE_MAX + +Internal timestamp retrieves from custom type are extracted in the following +format: 01-JAN-77 12.00.00.000000 AM. It is impossible to know the exact century +that must be used, so by default any year below 49 will be added to 2000 +and others to 1900. You can use this directive to change the default value 49. +this is only relevant if you have user defined type with a column timestamp. + +=item AUDIT_USER + +Set the comma separated list of username that must be used to filter +queries from the DBA_AUDIT_TRAIL table. Default is to not scan this +table and to never look for queries. This parameter is used only with +SHOW_REPORT and QUERY export type with no input file for queries. +Note that queries will be normalized before output unlike when a file +is given at input using the -i option or INPUT directive. + +=item FUNCTION_CHECK + +Disable this directive if you want to disable check_function_bodies. + + SET check_function_bodies = false; + +It disables validation of the function body string during CREATE FUNCTION. +Default is to use de postgresql.conf setting that enable it by default. + +=item ENABLE_BLOB_EXPORT + +Exporting BLOB takes time, in some circumstances you may want to export +all data except the BLOB columns. In this case disable this directive and +the BLOB columns will not be included into data export. Take care that the +target bytea column do not have a NOT NULL constraint. + +=item DATA_EXPORT_ORDER + +By default data export order will be done by sorting on table name. If you +have huge tables at end of alphabetic order and you are using multiprocess, +it can be better to set the sort order on size so that multiple small tables +can be processed before the largest tables finish. In this case set this +directive to size. Possible values are name and size. Note that export type +SHOW_TABLE and SHOW_COLUMN will use this sort order too, not only COPY or +INSERT export type. + +=back + +=head2 Limiting objects to export + +You may want to export only a part of an Oracle database, here are a set of +configuration directives that will allow you to control what parts of the +database should be exported. + +=over 4 + +=item ALLOW + +This directive allows you to set a list of objects on which the export must be +limited, excluding all other objects in the same type of export. The value is +a space or comma-separated list of objects name to export. You can include +valid regex into the list. For example: + + ALLOW EMPLOYEES SALE_.* COUNTRIES .*_GEOM_SEQ + +will export objects with name EMPLOYEES, COUNTRIES, all objects beginning with +'SALE_' and all objects with a name ending by '_GEOM_SEQ'. The object depends +of the export type. Note that regex will not works with 8i database, you must +use the % placeholder instead, Ora2Pg will use the LIKE operator. + +This is the manner to declare global filters that will be used with the current +export type. You can also use extended filters that will be applied on specific +objects or only on their related export type. For example: + + ora2pg -p -c ora2pg.conf -t TRIGGER -a 'TABLE[employees]' + +will limit export of trigger to those defined on table employees. If you want +to extract all triggers but not some INSTEAD OF triggers: + + ora2pg -c ora2pg.conf -t TRIGGER -e 'VIEW[trg_view_.*]' + +Or a more complex form: + + ora2pg -p -c ora2pg.conf -t TABLE -a 'TABLE[EMPLOYEES]' \ + -e 'INDEX[emp_.*];CKEY[emp_salary_min]' + +This command will export the definition of the employee table but will exclude +all index beginning with 'emp_' and the CHECK constraint called 'emp_salary_min'. + +When exporting partition you can exclude some partition tables by using + + ora2pg -p -c ora2pg.conf -t PARTITION -e 'PARTITION[PART_199.* PART_198.*]' + +This will exclude partitioned tables for year 1980 to 1999 from the export but +not the main partition table. The trigger will also be adapted to exclude those +table. + +With GRANT export you can use this extended form to exclude some users from the +export or limit the export to some others: + + ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' + +or + + ora2pg -p -c ora2pg.conf -t GRANT -a 'GRANT[USER1 USER2]' + +will limit export grants to users USER1 and USER2. But if you don't want to +export grants on some functions for these users, for example: + + ora2pg -p -c ora2pg.conf -t GRANT -a 'USER1 USER2' -e 'FUNCTION[adm_.*];PROCEDURE[adm_.*]' + +Advanced filters may need some learning. + +Oracle doesn't allow the use of lookahead expression so you may want to exclude +some object that match the ALLOW regexp you have defined. For example if you +want to export all table starting with E but not those starting with EXP it is +not possible to do that in a single expression. This is why you can start a +regular expression with the ! character to exclude object matching the regexp +given just after. Our previous example can be written as follow: + + ALLOW E.* !EXP.* + +it will be translated into: + + REGEXP_LIKE(..., '^E.*$') AND NOT REGEXP_LIKE(..., '^EXP.*$') + +in the object search expression. + +=item EXCLUDE + +This directive is the opposite of the previous, it allow you to define a space +or comma-separated list of object name to exclude from the export. You can +include valid regex into the list. For example: + + EXCLUDE EMPLOYEES TMP_.* COUNTRIES + +will exclude object with name EMPLOYEES, COUNTRIES and all tables beginning with +'tmp_'. + +For example, you can ban from export some unwanted function with this directive: + + EXCLUDE write_to_.* send_mail_.* + +this example will exclude all functions, procedures or functions in a package +with the name beginning with those regex. Note that regex will not work with +8i database, you must use the % placeholder instead, Ora2Pg will use the NOT +LIKE operator. + +See above (directive 'ALLOW') for the extended syntax. + + +=item VIEW_AS_TABLE + +Set which view to export as table. By default none. Value must be a list of +view name or regexp separated by space or comma. If the object name is a view +and the export type is TABLE, the view will be exported as a create table +statement. If export type is COPY or INSERT, the corresponding data will be +exported. + +See chapter "Exporting views as PostgreSQL table" for more details. + +=item NO_VIEW_ORDERING + +By default Ora2Pg try to order views to avoid error at import time with +nested views. With a huge number of views this can take a very long time, +you can bypass this ordering by enabling this directive. + +=item GRANT_OBJECT + +When exporting GRANTs you can specify a comma separated list of objects +for which privilege will be exported. Default is export for all objects. +Here are the possibles values TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, +PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, DIRECTORY. Only one object +type is allowed at a time. For example set it to TABLE if you just want to +export privilege on tables. You can use the -g option to overwrite it. + +When used this directive prevent the export of users unless it is set to USER. +In this case only users definitions are exported. + +=item WHERE + +This directive allows you to specify a WHERE clause filter when dumping the +contents of tables. Value is constructs as follows: TABLE_NAME[WHERE_CLAUSE], +or if you have only one where clause for each table just put the where clause +as the value. Both are possible too. Here are some examples: + + # Global where clause applying to all tables included in the export + WHERE 1=1 + + # Apply the where clause only on table TABLE_NAME + WHERE TABLE_NAME[ID1='001'] + + # Applies two different clause on tables TABLE_NAME and OTHER_TABLE + # and a generic where clause on DATE_CREATE to all other tables + WHERE TABLE_NAME[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' OTHER_TABLE[NAME='test'] + +Any where clause not included into a table name bracket clause will be applied +to all exported table including the tables defined in the where clause. These +WHERE clauses are very useful if you want to archive some data or at the +opposite only export some recent data. + +To be able to quickly test data import it is useful to limit data export to the +first thousand tuples of each table. For Oracle define the following clause: + + WHERE ROWNUM < 1000 + +and for MySQL, use the following: + + WHERE 1=1 LIMIT 1,1000 + +This can also be restricted to some tables data export. + +=item TOP_MAX + +This directive is used to limit the number of item shown in the top N lists +like the top list of tables per number of rows and the top list of largest +tables in megabytes. By default it is set to 10 items. + +=item LOG_ON_ERROR + +Enable this directive if you want to continue direct data import on error. +When Ora2Pg received an error in the COPY or INSERT statement from PostgreSQL +it will log the statement to a file called TABLENAME_error.log in the output +directory and continue to next bulk of data. Like this you can try to fix the +statement and manually reload the error log file. Default is disabled: abort +import on error. + +=item REPLACE_QUERY + +Sometime you may want to extract data from an Oracle table but you need a +custom query for that. Not just a "SELECT * FROM table" like Ora2Pg do +but a more complex query. This directive allows you to overwrite the query +used by Ora2Pg to extract data. The format is TABLENAME[SQL_QUERY]. +If you have multiple table to extract by replacing the Ora2Pg query, you can +define multiple REPLACE_QUERY lines. + + REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>'2014-08-01 00:00:00')] + +=back + +=head2 Control of Full Text Search export + +Several directives can be used to control the way Ora2Pg will export the +Oracle's Text search indexes. By default CONTEXT indexes will be exported +to PostgreSQL FTS indexes but CTXCAT indexes will be exported as indexes +using the pg_trgm extension. + +=over 4 + +=item CONTEXT_AS_TRGM + +Force Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using +pg_trgm extension. Default is to translate CONTEXT indexes into FTS indexes +and CTXCAT indexes using pg_trgm. Most of the time using pg_trgm is enough, +this is why this directive stand for. You need to create the pg_trgm extension +into the destination database before importing the objects: + + CREATE EXTENSION pg_trgm; + +=item FTS_INDEX_ONLY + +By default Ora2Pg creates a function-based index to translate Oracle Text +indexes. + + CREATE INDEX ON t_document + USING gin(to_tsvector('pg_catalog.french', title)); + +You will have to rewrite the CONTAIN() clause using to_tsvector(), example: + + SELECT id,title FROM t_document + WHERE to_tsvector(title)) @@ to_tsquery('search_word'); + +To force Ora2Pg to create an extra tsvector column with a dedicated triggers +for FTS indexes, disable this directive. In this case, Ora2Pg will add the +column as follow: ALTER TABLE t_document ADD COLUMN tsv_title tsvector; +Then update the column to compute FTS vectors if data have been loaded before + UPDATE t_document SET tsv_title = + to_tsvector('pg_catalog.french', coalesce(title,'')); +To automatically update the column when a modification in the title column +appears, Ora2Pg adds the following trigger: + + CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS $$ + BEGIN + IF TG_OP = 'INSERT' OR new.title != old.title THEN + new.tsv_title := + to_tsvector('pg_catalog.french', coalesce(new.title,'')); + END IF; + return new; + END + $$ LANGUAGE plpgsql; + CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE + ON t_document + FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); + +When the Oracle text index is defined over multiple column, Ora2Pg will use +setweight() to set a weight in the order of the column declaration. + +=item FTS_CONFIG + +Use this directive to force text search configuration to use. When it is not +set, Ora2Pg will autodetect the stemmer used by Oracle for each index and +pg_catalog.english if the information is not found. + + +=item USE_UNACCENT + +If you want to perform your text search in an accent insensitive way, enable +this directive. Ora2Pg will create an helper function over unaccent() and +creates the pg_trgm indexes using this function. With FTS Ora2Pg will +redefine your text search configuration, for example: + + CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); + ALTER TEXT SEARCH CONFIGURATION fr + ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; + +then set the FTS_CONFIG ora2pg.conf directive to fr instead of pg_catalog.english. + +When enabled, Ora2pg will create the wrapper function: + + CREATE OR REPLACE FUNCTION unaccent_immutable(text) + RETURNS text AS + $$ + SELECT public.unaccent('public.unaccent', $1); + $$ LANGUAGE sql IMMUTABLE + COST 1; + +the indexes are exported as follow: + + CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document + USING gin (unaccent_immutable(title) gin_trgm_ops); + +In your queries you will need to use the same function in the search to +be able to use the function-based index. Example: + + SELECT * FROM t_document + WHERE unaccent_immutable(title) LIKE '%donnees%'; + +=item USE_LOWER_UNACCENT + +Same as above but call lower() in the unaccent_immutable() function: + + CREATE OR REPLACE FUNCTION unaccent_immutable(text) + RETURNS text AS + $$ + SELECT lower(public.unaccent('public.unaccent', $1)); + $$ LANGUAGE sql IMMUTABLE; + + +=back + +=head2 Modifying object structure + +One of the great usage of Ora2Pg is its flexibility to replicate Oracle database +into PostgreSQL database with a different structure or schema. There's three +configuration directives that allow you to map those differences. + +=over 4 + +=item REORDERING_COLUMNS + +Enable this directive to reordering columns and minimized the footprint +on disc, so that more rows fit on a data page, which is the most important +factor for speed. Default is disabled, that mean the same order than in +Oracle tables definition, that's should be enough for most usage. This +directive is only used with TABLE export. + +=item MODIFY_STRUCT + +This directive allows you to limit the columns to extract for a given table. The +value consist in a space-separated list of table name with a set of column +between parenthesis as follow: + + MODIFY_STRUCT NOM_TABLE(nomcol1,nomcol2,...) ... + +for example: + + MODIFY_STRUCT T_TEST1(id,dossier) T_TEST2(id,fichier) + +This will only extract columns 'id' and 'dossier' from table T_TEST1 and columns +'id' and 'fichier' from the T_TEST2 table. This directive can only be used with +TABLE, COPY or INSERT export. With TABLE export create table DDL will respect +the new list of columns and all indexes or foreign key pointing to or from a +column removed will not be exported. + + +=item REPLACE_TABLES + +This directive allows you to remap a list of Oracle table name to a PostgreSQL table name during export. The value is a list of space-separated values with the following structure: + + REPLACE_TABLES ORIG_TBNAME1:DEST_TBNAME1 ORIG_TBNAME2:DEST_TBNAME2 + +Oracle tables ORIG_TBNAME1 and ORIG_TBNAME2 will be respectively renamed into +DEST_TBNAME1 and DEST_TBNAME2 + +=item REPLACE_COLS + +Like table name, the name of the column can be remapped to a different name +using the following syntax: + + REPLACE_COLS ORIG_TBNAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) + +For example: + + REPLACE_COLS T_TEST(dico:dictionary,dossier:folder) + +will rename Oracle columns 'dico' and 'dossier' from table T_TEST into new name +'dictionary' and 'folder'. + +=item REPLACE_AS_BOOLEAN + +If you want to change the type of some Oracle columns into PostgreSQL boolean +during the export you can define here a list of tables and column separated by +space as follow. + + REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 + +The values set in the boolean columns list will be replaced with the 't' and 'f' +following the default replacement values and those additionally set in directive +BOOLEAN_VALUES. + +Note that if you have modified the table name with REPLACE_TABLES and/or the +column's name, you need to use the name of the original table and/or column. + + REPLACE_COLS TB_NAME1(OLD_COL_NAME1:NEW_COL_NAME1) + REPLACE_AS_BOOLEAN TB_NAME1:OLD_COL_NAME1 + +You can also give a type and a precision to automatically convert all fields of +that type as a boolean. For example: + + REPLACE_AS_BOOLEAN NUMBER:1 CHAR:1 TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 + +will also replace any field of type number(1) or char(1) as a boolean in all exported +tables. + + +=item BOOLEAN_VALUES + +Use this to add additional definition of the possible boolean values used in +Oracle fields. You must set a space-separated list of TRUE:FALSE values. By +default here are the values recognized by Ora2Pg: + + BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled + +Any values defined here will be added to the default list. + +=item REPLACE_ZERO_DATE + +When Ora2Pg find a "zero" date: 0000-00-00 00:00:00 it is replaced by a NULL. +This could be a problem if your column is defined with NOT NULL constraint. +If you can not remove the constraint, use this directive to set an arbitral +date that will be used instead. You can also use -INFINITY if you don't want +to use a fake date. + +=item INDEXES_SUFFIX + +Add the given value as suffix to indexes names. Useful if you have indexes +with same name as tables. For example: + + INDEXES_SUFFIX _idx + +will add _idx at ed of all index name. Not so common but can help. + +=item INDEXES_RENAMING + +Enable this directive to rename all indexes using tablename_columns_names. +Could be very useful for database that have multiple time the same index name +or that use the same name than a table, which is not allowed by PostgreSQL +Disabled by default. + +=item USE_INDEX_OPCLASS + +Operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops +support B-tree indexes on the corresponding types. The difference from the +default operator classes is that the values are compared strictly character by +character rather than according to the locale-specific collation rules. This +makes these operator classes suitable for use by queries involving pattern +matching expressions (LIKE or POSIX regular expressions) when the database +does not use the standard "C" locale. If you enable, with value 1, this will +force Ora2Pg to export all indexes defined on varchar2() and char() columns +using those operators. If you set it to a value greater than 1 it will only +change indexes on columns where the character limit is greater or equal than +this value. For example, set it to 128 to create these kind of indexes on +columns of type varchar2(N) where N >= 128. + +=item PREFIX_PARTITION + +Enable this directive if you want that your partition table name will be +exported using the parent table name. Disabled by default. If you have +multiple partitioned table, when exported to PostgreSQL some partitions +could have the same name but different parent tables. This is not allowed, +table name must be unique. + + +=item PREFIX_SUB_PARTITION + +Enable this directive if you want that your subpartition table name will be +exported using the parent partition name. Enabled by default. If the partition +names are a part of the subpartition names, you should enable this directive. + + +=item DISABLE_PARTITION + +If you don't want to reproduce the partitioning like in Oracle and want to +export all partitioned Oracle data into the main single table in PostgreSQL +enable this directive. Ora2Pg will export all data into the main table name. +Default is to use partitioning, Ora2Pg will export data from each partition +and import them into the PostgreSQL dedicated partition table. + +=item DISABLE_UNLOGGED + +By default Ora2Pg export Oracle tables with the NOLOGGING attribute as +UNLOGGED tables. You may want to fully disable this feature because +you will lose all data from unlogged tables in case of a PostgreSQL crash. +Set it to 1 to export all tables as normal tables. + +=back + +=head2 Oracle Spatial to PostGis + +Ora2Pg fully export Spatial object from Oracle database. There's some +configuration directives that could be used to control the export. + +=over 4 + +=item AUTODETECT_SPATIAL_TYPE + +By default Ora2Pg is looking at indexes to see the spatial constraint type +and dimensions defined under Oracle. Those constraints are passed as at index +creation using for example: + + CREATE INDEX ... INDEXTYPE IS MDSYS.SPATIAL_INDEX + PARAMETERS('sdo_indx_dims=2, layer_gtype=point'); + +If those Oracle constraints parameters are not set, the default is to export +those columns as generic type GEOMETRY to be able to receive any spatial type. + +The AUTODETECT_SPATIAL_TYPE directive allows to force Ora2Pg to autodetect the +real spatial type and dimension used in a spatial column otherwise a non- +constrained "geometry" type is used. Enabling this feature will force Ora2Pg to +scan a sample of 50000 column to look at the GTYPE used. You can increase or +reduce the sample size by setting the value of AUTODETECT_SPATIAL_TYPE to the +desired number of line to scan. The directive is enabled by default. + +For example, in the case of a column named shape and defined with Oracle type +SDO_GEOMETRY, with AUTODETECT_SPATIAL_TYPE disabled it will be converted as: + + shape geometry(GEOMETRY) or shape geometry(GEOMETRYZ, 4326) + +and if the directive is enabled and the column just contains a single +geometry type that use a single dimension: + + shape geometry(POLYGON, 4326) or shape geometry(POLYGONZ, 4326) + +with a two or three dimensional polygon. + +=item CONVERT_SRID + +This directive allows you to control the automatically conversion of Oracle +SRID to standard EPSG. If enabled, Ora2Pg will use the Oracle function +sdo_cs.map_oracle_srid_to_epsg() to convert all SRID. Enabled by default. + +If the SDO_SRID returned by Oracle is NULL, it will be replaced by the +default value 8307 converted to its EPSG value: 4326 (see DEFAULT_SRID). + +If the value is upper than 1, all SRID will be forced to this value, in +this case DEFAULT_SRID will not be used when Oracle returns a null value +and the value will be forced to CONVERT_SRID. + +Note that it is also possible to set the EPSG value on Oracle side when +sdo_cs.map_oracle_srid_to_epsg() return NULL if your want to force the value: + + system@db> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; + +=item DEFAULT_SRID + +Use this directive to override the default EPSG SRID to used: 4326. +Can be overwritten by CONVERT_SRID, see above. + +=item GEOMETRY_EXTRACT_TYPE + +This directive can take three values: WKT (default), WKB and INTERNAL. +When it is set to WKT, Ora2Pg will use SDO_UTIL.TO_WKTGEOMETRY() to +extract the geometry data. When it is set to WKB, Ora2Pg will use the +binary output using SDO_UTIL.TO_WKBGEOMETRY(). If those two extract type +are calls at Oracle side, they are slow and you can easily reach Out Of +Memory when you have lot of rows. Also WKB is not able to export 3D geometry +and some geometries like CURVEPOLYGON. In this case you may use the INTERNAL +extraction type. It will use a Pure Perl library to convert the SDO_GEOMETRY +data into a WKT representation, the translation is done on Ora2Pg side. +This is a work in progress, please validate your exported data geometries +before use. Default spatial object extraction type is INTERNAL. + +=item POSTGIS_SCHEMA + +Use this directive to add a specific schema to the search path to look +for PostGis functions. + +=back + +=head2 PostgreSQL Import + +By default conversion to PostgreSQL format is written to file 'output.sql'. +The command: + + psql mydb < output.sql + +will import content of file output.sql into PostgreSQL mydb database. + +=over 4 + +=item DATA_LIMIT + +When you are performing INSERT/COPY export Ora2Pg proceed by chunks of DATA_LIMIT +tuples for speed improvement. Tuples are stored in memory before being written +to disk, so if you want speed and have enough system resources you can grow +this limit to an upper value for example: 100000 or 1000000. Before release 7.0 +a value of 0 mean no limit so that all tuples are stored in memory before being +flushed to disk. In 7.x branch this has been remove and chunk will be set to the +default: 10000 + +=item BLOB_LIMIT + +When Ora2Pg detect a table with some BLOB it will automatically reduce the +value of this directive by dividing it by 10 until his value is below 1000. +You can control this value by setting BLOB_LIMIT. Exporting BLOB use lot of +resources, setting it to a too high value can produce OOM. + +=item OUTPUT + +The Ora2Pg output filename can be changed with this directive. Default value is +output.sql. if you set the file name with extension .gz or .bz2 the output will +be automatically compressed. This require that the Compress::Zlib Perl module +is installed if the filename extension is .gz and that the bzip2 system command +is installed for the .bz2 extension. + +=item OUTPUT_DIR + +Since release 7.0, you can define a base directory where the file will be written. +The directory must exists. + +=item BZIP2 + +This directive allows you to specify the full path to the bzip2 program if it +can not be found in the PATH environment variable. + +=item FILE_PER_CONSTRAINT + +Allow object constraints to be saved in a separate file during schema export. +The file will be named CONSTRAINTS_OUTPUT, where OUTPUT is the value of the +corresponding configuration directive. You can use .gz xor .bz2 extension to +enable compression. Default is to save all data in the OUTPUT file. This +directive is usable only with TABLE export type. + +The constraints can be imported quickly into PostgreSQL using the LOAD export +type to parallelize their creation over multiple (-j or JOBS) connections. + +=item FILE_PER_INDEX + +Allow indexes to be saved in a separate file during schema export. The file +will be named INDEXES_OUTPUT, where OUTPUT is the value of the corresponding +configuration directive. You can use .gz xor .bz2 file extension to enable +compression. Default is to save all data in the OUTPUT file. This directive +is usable only with TABLE AND TABLESPACE export type. With the TABLESPACE +export, it is used to write "ALTER INDEX ... TABLESPACE ..." into a separate +file named TBSP_INDEXES_OUTPUT that can be loaded at end of the migration after +the indexes creation to move the indexes. + +The indexes can be imported quickly into PostgreSQL using the LOAD export +type to parallelize their creation over multiple (-j or JOBS) connections. + +=item FILE_PER_FKEYS + +Allow foreign key declaration to be saved in a separate file during +schema export. By default foreign keys are exported into the main +output file or in the CONSTRAINT_output.sql file. When enabled foreign +keys will be exported into a file named FKEYS_output.sql + +=item FILE_PER_TABLE + +Allow data export to be saved in one file per table/view. The files will be +named as tablename_OUTPUT, where OUTPUT is the value of the corresponding +configuration directive. You can still use .gz xor .bz2 extension in the OUTPUT +directive to enable compression. Default 0 will save all data in one file, set +it to 1 to enable this feature. This is usable only during INSERT or COPY export +type. + +=item FILE_PER_FUNCTION + +Allow functions, procedures and triggers to be saved in one file per object. +The files will be named as objectname_OUTPUT. Where OUTPUT is the value of the +corresponding configuration directive. You can still use .gz xor .bz2 extension +in the OUTPUT directive to enable compression. Default 0 will save all in one +single file, set it to 1 to enable this feature. This is usable only during the +corresponding export type, the package body export has a special behavior. + +When export type is PACKAGE and you've enabled this directive, Ora2Pg will +create a directory per package, named with the lower case name of the package, +and will create one file per function/procedure into that directory. If the +configuration directive is not enabled, it will create one file per package as +packagename_OUTPUT, where OUTPUT is the value of the corresponding directive. + +=item TRUNCATE_TABLE + +If this directive is set to 1, a TRUNCATE TABLE instruction will be add before +loading data. This is usable only during INSERT or COPY export type. + +When activated, the instruction will be added only if there's no global DELETE +clause or not one specific to the current table (see below). + +=item DELETE + +Support for include a DELETE FROM ... WHERE clause filter before importing +data and perform a delete of some lines instead of truncating tables. +Value is construct as follow: TABLE_NAME[DELETE_WHERE_CLAUSE], or +if you have only one where clause for all tables just put the delete +clause as single value. Both are possible too. Here are some examples: + + DELETE 1=1 # Apply to all tables and delete all tuples + DELETE TABLE_TEST[ID1='001'] # Apply only on table TABLE_TEST + DELETE TABLE_TEST[ID1='001' OR ID1='002] DATE_CREATE > '2001-01-01' TABLE_INFO[NAME='test'] + +The last applies two different delete where clause on tables TABLE_TEST and +TABLE_INFO and a generic delete where clause on DATE_CREATE to all other tables. +If TRUNCATE_TABLE is enabled it will be applied to all tables not covered by +the DELETE definition. + +These DELETE clauses might be useful with regular "updates". + +=item STOP_ON_ERROR + +Set this parameter to 0 to not include the call to \set ON_ERROR_STOP ON in +all SQL scripts generated by Ora2Pg. By default this order is always present +so that the script will immediately abort when an error is encountered. + +=item COPY_FREEZE + +Enable this directive to use COPY FREEZE instead of a simple COPY to +export data with rows already frozen. This is intended as a performance +option for initial data loading. Rows will be frozen only if the table +being loaded has been created or truncated in the current sub-transaction. +This will only work with export to file and when -J or ORACLE_COPIES is +not set or default to 1. It can be used with direct import into PostgreSQL +under the same condition but -j or JOBS must also be unset or default to 1. + +=item CREATE_OR_REPLACE + +By default Ora2Pg uses CREATE OR REPLACE in function DDL, if you need not +to override existing functions disable this configuration directive, +DDL will not include OR REPLACE. + +=item NO_HEADER + +Enabling this directive will prevent Ora2Pg to print his header into +output files. Only the translated code will be written. + +=item PSQL_RELATIVE_PATH + +By default Ora2Pg use \i psql command to execute generated SQL files +if you want to use a relative path following the script execution file +enabling this option will use \ir. See psql help for more information. + +=back + +When using Ora2Pg export type INSERT or COPY to dump data to file and that +FILE_PER_TABLE is enabled, you will be warned that Ora2Pg will not export +data again if the file already exists. This is to prevent downloading twice +table with huge amount of data. To force the download of data from these tables +you have to remove the existing output file first. + +If you want to import data on the fly to the PostgreSQL database you have three +configuration directives to set the PostgreSQL database connection. This is only +possible with COPY or INSERT export type as for database schema there's no real +interest to do that. + +=over 4 + +=item PG_DSN + +Use this directive to set the PostgreSQL data source namespace using DBD::Pg +Perl module as follow: + + dbi:Pg:dbname=pgdb;host=localhost;port=5432 + +will connect to database 'pgdb' on localhost at tcp port 5432. + +Note that this directive is only used for data export, other export need to +be imported manually through the use og psql or any other PostgreSQL client. + +=item PG_USER and PG_PWD + +These two directives are used to set the login user and password. + +If you do not supply a credential with PG_PWD and you have installed the +Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If +PG_USER is not set it will be asked interactively too. + + +=item SYNCHRONOUS_COMMIT + +Specifies whether transaction commit will wait for WAL records to be written +to disk before the command returns a "success" indication to the client. This +is the equivalent to set synchronous_commit directive of postgresql.conf file. +This is only used when you load data directly to PostgreSQL, the default is +off to disable synchronous commit to gain speed at writing data. Some modified +version of PostgreSQL, like greenplum, do not have this setting, so in this +set this directive to 1, ora2pg will not try to change the setting. + +=item PG_INITIAL_COMMAND + +This directive can be used to send an initial command to PostgreSQL, just after +the connection. For example to set some session parameters. This directive can +be used multiple times. + +=back + +=head2 Column type control + +=over 4 + +=item PG_NUMERIC_TYPE + +If set to 1 replace portable numeric type into PostgreSQL internal type. +Oracle data type NUMBER(p,s) is approximatively converted to real and +float PostgreSQL data type. If you have monetary fields or don't want +rounding issues with the extra decimals you should preserve the same +numeric(p,s) PostgreSQL data type. Do that only if you need exactness +because using numeric(p,s) is slower than using real or double. + +=item PG_INTEGER_TYPE + +If set to 1 replace portable numeric type into PostgreSQL internal type. +Oracle data type NUMBER(p) or NUMBER are converted to smallint, integer +or bigint PostgreSQL data type following the value of the precision. If +NUMBER without precision are set to DEFAULT_NUMERIC (see below). + +=item DEFAULT_NUMERIC + +NUMBER without precision are converted by default to bigint only if +PG_INTEGER_TYPE is true. You can overwrite this value to any PG type, +like integer or float. + +=item DATA_TYPE + +If you're experiencing any problem in data type schema conversion with this +directive you can take full control of the correspondence between Oracle and +PostgreSQL types to redefine data type translation used in Ora2pg. The syntax +is a comma-separated list of "Oracle datatype:Postgresql datatype". Here are +the default list used: + + DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,DATE:timestamp,LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:numeric,INTEGER:numeric,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone + +Note that the directive and the list definition must be a single line. + +If you want to replace a type with a precision and scale you need to escape +the coma with a backslash. For example, if you want to replace all NUMBER(*,0) +into bigint instead of numeric(38) add the following: + + DATA_TYPE NUMBER(*\,0):bigint + +You don't have to recopy all default type conversion but just the one you want +to rewrite. + +There's a special case with BFILE when they are converted to type TEXT, they +will just contains the full path to the external file. If you set the +destination type to BYTEA, the default, Ora2Pg will export the content of the +BFILE as bytea. The third case is when you set the destination type to EFILE, +in this case, Ora2Pg will export it as an EFILE record: (DIRECTORY, FILENAME). +Use the DIRECTORY export type to export the existing directories as well as +privileges on those directories. + + +There's no SQL function available to retrieve the path to the BFILE. Ora2Pg +have to create one using the DBMS_LOB package. + + CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) + RETURN VARCHAR2 + AS + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + l_path VARCHAR2(4000); + BEGIN + dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); + SELECT directory_path INTO l_path FROM all_directories + WHERE directory_name = l_dir; + l_dir := rtrim(l_path,'/'); + RETURN l_dir || '/' || l_fname; + END; + +This function is only created if Ora2Pg found a table with a BFILE column and +that the destination type is TEXT. The function is dropped at the end of the +export. This concern both, COPY and INSERT export type. + +There's no SQL function available to retrieve BFILE as an EFILE record, then +Ora2Pg have to create one using the DBMS_LOB package. + + CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) + RETURN VARCHAR2 + AS + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + BEGIN + dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); + RETURN '(' || l_dir || ',' || l_fnamei || ')'; + END; + +This function is only created if Ora2Pg found a table with a BFILE column and +that the destination type is EFILE. The function is dropped at the end of the +export. This concern both, COPY and INSERT export type. + +To set the destination type, use the DATA_TYPE configuration directive: + + DATA_TYPE BFILE:EFILE + +for example. + +The EFILE type is a user defined type created by the PostgreSQL extension +external_file that can be found here: https://github.com/darold/external_file +This is a port of the BFILE Oracle type to PostgreSQL. + +There's no SQL function available to retrieve the content of a BFILE. Ora2Pg +have to create one using the DBMS_LOB package. + + CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN + BLOB + AS + filecontent BLOB := NULL; + src_file BFILE := NULL; + l_step PLS_INTEGER := 12000; + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + offset NUMBER := 1; + BEGIN + IF p_bfile IS NULL THEN + RETURN NULL; + END IF; + + DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); + src_file := BFILENAME( l_dir, l_fname ); + IF src_file IS NULL THEN + RETURN NULL; + END IF; + + DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); + DBMS_LOB.CREATETEMPORARY(filecontent, true); + DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); + DBMS_LOB.FILECLOSE(src_file); + RETURN filecontent; + END; + +This function is only created if Ora2Pg found a table with a BFILE column and +that the destination type is bytea (the default). The function is dropped at +the end of the export. This concern both, COPY and INSERT export type. + +About the ROWID and UROWID, they are converted into OID by "logical" default +but this will through an error at data import. There is no equivalent data type +so you might want to use the DATA_TYPE directive to change the corresponding +type in PostgreSQL. You should consider replacing this data type by a bigserial +(autoincremented sequence), text or uuid data type. + + +=item MODIFY_TYPE + +Sometimes you need to force the destination type, for example a column +exported as timestamp by Ora2Pg can be forced into type date. Value is +a comma-separated list of TABLE:COLUMN:TYPE structure. If you need to use +comma or space inside type definition you will have to backslash them. + + MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\,6) + +Type of table1.col3 will be replaced by a varchar and table1.col4 by +a decimal with precision and scale. + +If the column's type is a user defined type Ora2Pg will autodetect the +composite type and will export its data using ROW(). Some Oracle user +defined types are just array of a native type, in this case you may want +to transform this column in simple array of a PostgreSQL native type. +To do so, just redefine the destination type as wanted and Ora2Pg will +also transform the data as an array. For example, with the following +definition in Oracle: + + CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); + CREATE TABLE club (Name VARCHAR2(10), + Address VARCHAR2(20), + City VARCHAR2(20), + Phone VARCHAR2(8), + Members mem_type + ); + +custom type "mem_type" is just a string array and can be translated into +the following in PostgreSQL: + + CREATE TABLE club ( + name varchar(10), + address varchar(20), + city varchar(20), + phone varchar(8), + members text[] + ) ; + +To do so, just use the directive as follow: + + MODIFY_TYPE CLUB:MEMBERS:text[] + +Ora2Pg will take care to transform all data of this column in the correct +format. Only arrays of characters and numerics types are supported. + +=back + +=head2 Taking export under control + +The following other configuration directives interact directly with the export process and give you fine granularity in database export control. + +=over 4 + +=item SKIP + +For TABLE export you may not want to export all schema constraints, the SKIP +configuration directive allows you to specify a space-separated list of +constraints that should not be exported. Possible values are: + + - fkeys: turn off foreign key constraints + - pkeys: turn off primary keys + - ukeys: turn off unique column constraints + - indexes: turn off all other index types + - checks: turn off check constraints + +For example: + + SKIP indexes,checks + +will removed indexes and check constraints from export. + +=item PKEY_IN_CREATE + +Enable this directive if you want to add primary key definition inside the +create table statement. If disabled (the default) primary key definition +will be added with an alter table statement. Enable it if you are exporting +to GreenPlum PostgreSQL database. + +=item KEEP_PKEY_NAMES + +By default names of the primary and unique key in the source Oracle database +are ignored and key names are autogenerated in the target PostgreSQL database +with the PostgreSQL internal default naming rules. If you want to preserve +Oracle primary and unique key names set this option to 1. + +=item FKEY_ADD_UPDATE + +This directive allows you to add an ON UPDATE CASCADE option to a foreign +key when a ON DELETE CASCADE is defined or always. Oracle do not support +this feature, you have to use trigger to operate the ON UPDATE CASCADE. +As PostgreSQL has this feature, you can choose how to add the foreign +key option. There are three values to this directive: never, the default +that mean that foreign keys will be declared exactly like in Oracle. +The second value is delete, that mean that the ON UPDATE CASCADE option +will be added only if the ON DELETE CASCADE is already defined on the +foreign Keys. The last value, always, will force all foreign keys to be +defined using the update option. + +=item FKEY_DEFERRABLE + +When exporting tables, Ora2Pg normally exports constraints as they are, if they +are non-deferrable they are exported as non-deferrable. However, non-deferrable +constraints will probably cause problems when attempting to import data to Pg. +The FKEY_DEFERRABLE option set to 1 will cause all foreign key constraints to +be exported as deferrable. + +=item DEFER_FKEY + +In addition to exporting data when the DEFER_FKEY option set to 1, it will add +a command to defer all foreign key constraints during data export and +the import will be done in a single transaction. This will work only if +foreign keys have been exported as deferrable and you are not using direct +import to PostgreSQL (PG_DSN is not defined). Constraints will then be +checked at the end of the transaction. + +This directive can also be enabled if you want to force all foreign keys +to be created as deferrable and initially deferred during schema export +(TABLE export type). + +=item DROP_FKEY + +If deferring foreign keys is not possible due to the amount of data in a +single transaction, you've not exported foreign keys as deferrable or you +are using direct import to PostgreSQL, you can use the DROP_FKEY directive. + +It will drop all foreign keys before all data import and recreate them at +the end of the import. + +=item DROP_INDEXES + +This directive allows you to gain lot of speed improvement during data import +by removing all indexes that are not an automatic index (indexes of primary +keys) and recreate them at the end of data import. Of course it is far better +to not import indexes and constraints before having imported all data. + +=item DISABLE_TRIGGERS + +This directive is used to disable triggers on all tables in COPY or INSERT +export modes. Available values are USER (disable user-defined triggers only) +and ALL (includes RI system triggers). Default is 0: do not add SQL statements +to disable trigger before data import. + +If you want to disable triggers during data migration, set the value to +USER if your are connected as non superuser and ALL if you are connected +as PostgreSQL superuser. A value of 1 is equal to USER. + +=item DISABLE_SEQUENCE + +If set to 1 it disables alter of sequences on all tables during COPY or INSERT export +mode. This is used to prevent the update of sequence during data migration. +Default is 0, alter sequences. + +=item NOESCAPE + +By default all data that are not of type date or time are escaped. If you +experience any problem with that you can set it to 1 to disable character +escaping during data export. This directive is only used during a COPY export. +See STANDARD_CONFORMING_STRINGS for enabling/disabling escape with INSERT +statements. + +=item STANDARD_CONFORMING_STRINGS + +This controls whether ordinary string literals ('...') treat backslashes +literally, as specified in SQL standard. This was the default before Ora2Pg +v8.5 so that all strings was escaped first, now this is currently on, causing +Ora2Pg to use the escape string syntax (E'...') if this parameter is not +set to 0. This is the exact behavior of the same option in PostgreSQL. +This directive is only used during data export to build INSERT statements. +See NOESCAPE for enabling/disabling escape in COPY statements. + +=item TRIM_TYPE + +If you want to convert CHAR(n) from Oracle into varchar(n) or text on PostgreSQL +using directive DATA_TYPE, you might want to do some trimming on the data. By +default Ora2Pg will auto-detect this conversion and remove any whitespace at both +leading and trailing position. If you just want to remove the leadings character +set the value to LEADING. If you just want to remove the trailing character, set +the value to TRAILING. Default value is BOTH. + +=item TRIM_CHAR + +The default trimming character is space, use this directive if you need to +change the character that will be removed. For example, set it to - if you +have leading - in the char(n) field. To use space as trimming charger, comment +this directive, this is the default value. + +=item PRESERVE_CASE + +If you want to preserve the case of Oracle object name set this directive to 1. +By default Ora2Pg will convert all Oracle object names to lower case. I do not +recommend to enable this unless you will always have to double-quote object +names on all your SQL scripts. + +=item ORA_RESERVED_WORDS + +Allow escaping of column name using Oracle reserved words. Value is a list of +comma-separated reserved word. Default: audit,comment,references. + +=item USE_RESERVED_WORDS + +Enable this directive if you have table or column names that are a reserved +word for PostgreSQL. Ora2Pg will double quote the name of the object. + +=item GEN_USER_PWD + +Set this directive to 1 to replace default password by a random password for all +extracted user during a GRANT export. + +=item PG_SUPPORTS_MVIEW + +Since PostgreSQL 9.3, materialized view are supported with the SQL syntax +'CREATE MATERIALIZED VIEW'. To force Ora2Pg to use the native PostgreSQL +support you must enable this configuration - enable by default. If you want +to use the old style with table and a set of function, you should disable it. + +=item PG_SUPPORTS_IFEXISTS + +PostgreSQL version below 9.x do not support IF EXISTS in DDL statements. +Disabling the directive with value 0 will prevent Ora2Pg to add those +keywords in all generated statements. Default value is 1, enabled. + +=item PG_SUPPORTS_ROLE (Deprecated) + +This option is deprecated since Ora2Pg release v7.3. + +By default Oracle roles are translated into PostgreSQL groups. If you have +PostgreSQL 8.1 or more consider the use of ROLES and set this directive to 1 +to export roles. + +=item PG_SUPPORTS_INOUT (Deprecated) + +This option is deprecated since Ora2Pg release v7.3. + +If set to 0, all IN, OUT or INOUT parameters will not be used into the generated +PostgreSQL function declarations (disable it for PostgreSQL database version +lower than 8.1), This is now enable by default. + +=item PG_SUPPORTS_DEFAULT + +This directive enable or disable the use of default parameter value in function +export. Until PostgreSQL 8.4 such a default value was not supported, this feature +is now enable by default. + +=item PG_SUPPORTS_WHEN (Deprecated) + +Add support to WHEN clause on triggers as PostgreSQL v9.0 now support it. This +directive is enabled by default, set it to 0 disable this feature. + +=item PG_SUPPORTS_INSTEADOF (Deprecated) + +Add support to INSTEAD OF usage on triggers (used with PG >= 9.1), if this +directive is disabled the INSTEAD OF triggers will be rewritten as Pg rules. + +=item PG_SUPPORTS_CHECKOPTION + +When enabled, export views with CHECK OPTION. Disable it if you have PostgreSQL +version prior to 9.4. Default: 1, enabled. + +=item PG_SUPPORTS_IFEXISTS + +If disabled, do not export object with IF EXISTS statements. +Enabled by default. + +=item PG_SUPPORTS_PARTITION + +PostgreSQL version prior to 10.0 do not have native partitioning. +Enable this directive if you want to use declarative partitioning. +Enable by default. + +=item PG_SUPPORTS_SUBSTR + +Some versions of PostgreSQL like Redshift doesn't support substr() +and it need to be replaced by a call to substring(). In this case, +disable it. + +=item PG_SUPPORTS_NAMED_OPERATOR + +Disable this directive if you are using PG < 9.5, PL/SQL operator used in +named parameter => will be replaced by PostgreSQL proprietary operator := +Enable by default. + +=item PG_SUPPORTS_IDENTITY + +Enable this directive if you have PostgreSQL >= 10 to use IDENTITY columns +instead of serial or bigserial data type. If PG_SUPPORTS_IDENTITY is disabled +and there is IDENTITY column in the Oracle table, they are exported as serial +or bigserial columns. When it is enabled they are exported as IDENTITY columns +like: + + CREATE TABLE identity_test_tab ( + id bigint GENERATED ALWAYS AS IDENTITY, + description varchar(30) + ) ; + +If there is non default sequence options set in Oracle, they will be appended +after the IDENTITY keyword. +Additionally in both cases, Ora2Pg will create a file AUTOINCREMENT_output.sql +with a embedded function to update the associated sequences with the restart +value set to "SELECT max(colname)+1 FROM tablename". Of course this file must +be imported after data import otherwise sequence will be kept to start value. +Enabled by default. + +=item PG_SUPPORTS_PROCEDURE + +PostgreSQL v11 adds support of PROCEDURE, enable it if you use such version. + +=item BITMAP_AS_GIN + +Use btree_gin extension to create bitmap like index with pg >= 9.4 +You will need to create the extension by yourself: + create extension btree_gin; +Default is to create GIN index, when disabled, a btree index will be created + +=item PG_BACKGROUND + +Use pg_background extension to create an autonomous transaction instead +of using a dblink wrapper. With pg >= 9.5 only. Default is to use dblink. +See https://github.com/vibhorkum/pg_background about this extension. + +=item DBLINK_CONN + +By default if you have an autonomous transaction translated using dblink +extension instead of pg_background the connection is defined using the +values set with PG_DSN, PG_USER and PG_PWD. If you want to fully override +the connection string use this directive as follow to set the connection +in the autonomous transaction wrapper function. For example: + + DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass + +=item LONGREADLEN + +Use this directive to set the database handle's 'LongReadLen' attribute to a +value that will be the larger than the expected size of the LOBs. The default +is 1MB witch may not be enough to extract BLOBs or CLOBs. If the size of the +LOB exceeds the 'LongReadLen' DBD::Oracle will return a 'ORA-24345: A Truncation' +error. Default: 1023*1024 bytes. + +Take a look at this page to learn more: http://search.cpan.org/~pythian/DBD-Oracle-1.22/Oracle.pm#Data_Interface_for_Persistent_LOBs + +Important note: If you increase the value of this directive take care that +DATA_LIMIT will probably needs to be reduced. Even if you only have a 1MB blob, +trying to read 10000 of them (the default DATA_LIMIT) all at once will require +10GB of memory. You may extract data from those table separately and set a +DATA_LIMIT to 500 or lower, otherwise you may experience some out of memory. + +=item LONGTRUNKOK + +If you want to bypass the 'ORA-24345: A Truncation' error, set this directive +to 1, it will truncate the data extracted to the LongReadLen value. Disable +by default so that you will be warned if your LongReadLen value is not high +enough. + +=item USE_LOB_LOCATOR + +Disable this if you want to load full content of BLOB and CLOB and not use +LOB locators. In this case you will have to set LONGREADLEN to the right +value. Note that this will not improve speed of BLOB export as most of the +time is always consumed by the bytea escaping and in this case export is +done line by line and not by chunk of DATA_LIMIT rows. For more information +on how it works, see http://search.cpan.org/~pythian/DBD-Oracle-1.74/lib/DBD/Oracle.pm#Data_Interface_for_LOB_Locators + +Default is enabled, it use LOB locators. + +=item LOB_CHUNK_SIZE + +Oracle recommends reading from and writing to a LOB in batches using a +multiple of the LOB chunk size. This chunk size defaults to 8k (8192). +Recent tests shown that the best performances can be reach with higher +value like 512K or 4Mb. + +A quick benchmark with 30120 rows with different size of BLOB (200x5Mb, +19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with DATA_LIMIT=100, +LONGREADLEN=170Mb and a total table size of 20GB gives: + + no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) + chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) + chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) + chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) + +In conclusion it can be more than 10 time faster with LOB_CHUNK_SIZE set +to 4Mb. Depending of the size of most BLOB you may want to adjust the value +here. For example if you have a majority of small lobs bellow 8K, using 8192 +is better to not waste space. Default value for LOB_CHUNK_SIZE is 512000. + +=item XML_PRETTY + +Force the use getStringVal() instead of getClobVal() for XML data export. Default is 1, +enabled for backward compatibility. Set it to 0 to use extract method a la CLOB. +Note that XML value extracted with getStringVal() must not exceed VARCHAR2 size +limit (4000) otherwise it will return an error. + +=item ENABLE_MICROSECOND + +Set it to O if you want to disable export of millisecond from Oracle timestamp +columns. By default milliseconds are exported with the use of following format: + + 'YYYY-MM-DD HH24:MI:SS.FF' + +Disabling will force the use of the following Oracle format: + + to_char(..., 'YYYY-MM-DD HH24:MI:SS') + +By default milliseconds are exported. + +=item DISABLE_COMMENT + +Set this to 1 if you don't want to export comment associated to tables and +columns definition. Default is enabled. + +=back + +=head2 Control MySQL export behavior + +=over 4 + +=item MYSQL_PIPES_AS_CONCAT + +Enable this if double pipe and double ampersand (|| and &&) should not be +taken as equivalent to OR and AND. It depend of the variable @sql_mode, +Use it only if Ora2Pg fail on auto detecting this behavior. + +=item MYSQL_INTERNAL_EXTRACT_FORMAT + +Enable this directive if you want EXTRACT() replacement to use the internal +format returned as an integer, for example DD HH24:MM:SS will be replaced +with format; DDHH24MMSS::bigint, this depend of your apps usage. + +=back + +=head2 Special options to handle character encoding + +=over 4 + +=item NLS_LANG and NLS_NCHAR + +By default Ora2Pg will set NLS_LANG to AMERICAN_AMERICA.AL32UTF8 and NLS_NCHAR +to AL32UTF8. It is not recommended to change those settings but in some case it +could be useful. Using your own settings with those configuration directive will +change the client encoding at Oracle side by setting the environment variables +$ENV{NLS_LANG} and $ENV{NLS_NCHAR}. + +=item BINMODE + +By default Ora2Pg will force Perl to use utf8 I/O encoding. This is done through +a call to the Perl pragma: + + use open ':utf8'; + +You can override this encoding by using the BINMODE directive, for example you +can set it to :locale to use your locale or iso-8859-7, it will respectively use + + use open ':locale'; + use open ':encoding(iso-8859-7)'; + +If you have change the NLS_LANG in non UTF8 encoding, you might want to set this +directive. See http://perldoc.perl.org/5.14.2/open.html for more information. +Most of the time, leave this directive commented. + +=item CLIENT_ENCODING + +By default PostgreSQL client encoding is automatically set to UTF8 to avoid +encoding issue. If you have changed the value of NLS_LANG you might have to +change the encoding of the PostgreSQL client. + +You can take a look at the PostgreSQL supported character sets here: http://www.postgresql.org/docs/9.0/static/multibyte.html + +=back + +=head2 PLSQL to PLPGSQL conversion + +Automatic code conversion from Oracle PLSQL to PostgreSQL PLPGSQL is a work in +progress in Ora2Pg and surely you will always have manual work. The Perl code +used for automatic conversion is all stored in a specific Perl Module named +Ora2Pg/PLSQL.pm feel free to modify/add you own code and send me patches. The +main work in on function, procedure, package and package body headers and +parameters rewrite. + +=over 4 + +=item PLSQL_PGSQL + +Enable/disable PLSQL to PLPGSQL conversion. Enabled by default. + +=item NULL_EQUAL_EMPTY + +Ora2Pg can replace all conditions with a test on NULL by a call to the +coalesce() function to mimic the Oracle behavior where empty string are +considered equal to NULL. + + (field1 IS NULL) is replaced by (coalesce(field1::text, '') = '') + (field2 IS NOT NULL) is replaced by (field2 IS NOT NULL AND field2::text <> '') + +You might want this replacement to be sure that your application will have the +same behavior but if you have control on you application a better way is to +change it to transform empty string into NULL because PostgreSQL makes the +difference. + +=item EMPTY_LOB_NULL + +Force empty_clob() and empty_blob() to be exported as NULL instead as empty +string for the first one and '\x' for the second. If NULL is allowed in your +column this might improve data export speed if you have lot of empty lob. +Default is to preserve the exact data from Oracle. + +=item PACKAGE_AS_SCHEMA + +If you don't want to export package as schema but as simple functions you +might also want to replace all call to package_name.function_name. If you +disable the PACKAGE_AS_SCHEMA directive then Ora2Pg will replace all call +to package_name.function_name() by package_name_function_name(). Default +is to use a schema to emulate package. + +The replacement will be done in all kind of DDL or code that is parsed by +the PLSQL to PLPGSQL converter. PLSQL_PGSQL must be enabled or -p used in +command line. + +=item REWRITE_OUTER_JOIN + +Enable this directive if the rewrite of Oracle native syntax (+) of +OUTER JOIN is broken. This will force Ora2Pg to not rewrite such code, +default is to try to rewrite simple form of right outer join for the +moment. + +=item UUID_FUNCTION + +By default Ora2Pg will convert call to SYS_GUID() Oracle function +with a call to uuid_generate_v4 from uuid-ossp extension. You can +redefined it to use the gen_random_uuid function from pgcrypto +extension by changing the function name. Default to uuid_generate_v4. + +Note that when a RAW(n) column has "SYS_GUID()" as default value +Ora2Pg will automatically translate the type of the column into uuid +which might be the right translation in most of the case. + +=item FUNCTION_STABLE + +By default Oracle functions are marked as STABLE as they can not modify data +unless when used in PL/SQL with variable assignment or as conditional +expression. You can force Ora2Pg to create these function as VOLATILE by +disabling this configuration directive. + +=item COMMENT_COMMIT_ROLLBACK + +By default call to COMMIT/ROLLBACK are kept untouched by Ora2Pg to force +the user to review the logic of the function. Once it is fixed in Oracle +source code or you want to comment this calls enable the following directive. + +=item COMMENT_SAVEPOINT + +It is common to see SAVEPOINT call inside PL/SQL procedure together with +a ROLLBACK TO savepoint_name. When COMMENT_COMMIT_ROLLBACK is enabled you +may want to also comment SAVEPOINT calls, in this case enable it. + +=item STRING_CONSTANT_REGEXP + +Ora2Pg replace all string constant during the pl/sql to plpgsql translation, +string constant are all text include between single quote. If you have some +string placeholder used in dynamic call to queries you can set a list of +regexp to be temporary replaced to not break the parser. For example: + + STRING_CONSTANT_REGEXP + +The list of regexp must use the semi colon as separator. + +=item ALTERNATIVE_QUOTING_REGEXP + +To support the Alternative Quoting Mechanism ('Q' or 'q') for String Literals +set the regexp with the text capture to use to extract the text part. For +example with a variable declared as + + c_sample VARCHAR2(100 CHAR) := q'{This doesn't work.}'; + +the regexp to use must be: + + ALTERNATIVE_QUOTING_REGEXP q'{(.*)}' + +ora2pg will use the $$ delimiter, with the example the result will be: + + c_sample varchar(100) := $$This doesn't work.$$; + +The value of this configuration directive can be a list of regexp +separated by a semi colon. The capture part (between parenthesis) is +mandatory in each regexp if you want to restore the string constant. + + +=item USE_ORAFCE + +If you want to use functions defined in the Orafce library and prevent +Ora2Pg to translate call to these functions, enable this directive. +The Orafce library can be found here: https://github.com/orafce/orafce + +By default Ora2pg rewrite add_month(), add_year(), date_trunc() and +to_char() functions, but you may prefer to use the orafce version of +these function that do not need any code transformation. + +=item AUTONOMOUS_TRANSACTION + +Enable translation of autonomous transactions into a wrapper function +using dblink or pg_background extension. If you don't want to use this +translation and just want the function to be exported as a normal one +without the pragma call, disable this directive. + +=back + +=head2 Materialized view + +Materialized views are exported as snapshot "Snapshot Materialized Views" as +PostgreSQL only supports full refresh. + +If you want to import the materialized views in PostgreSQL prior to 9.3 you +have to set configuration directive PG_SUPPORTS_MVIEW to 0. In this case +Ora2Pg will export all materialized views as explain in this document: + + http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views. + +When exporting materialized view Ora2Pg will first add the SQL code to create the "materialized_views" table: + + CREATE TABLE materialized_views ( + mview_name text NOT NULL PRIMARY KEY, + view_name text NOT NULL, + iname text, + last_refresh TIMESTAMP WITH TIME ZONE + ); + +all materialized views will have an entry in this table. It then adds the +plpgsql code to create tree functions: + + create_materialized_view(text, text, text) used to create a materialized view + drop_materialized_view(text) used to delete a materialized view + refresh_full_materialized_view(text) used to refresh a view + +then it adds the SQL code to create the view and the materialized view: + + CREATE VIEW mviewname_mview AS + SELECT ... FROM ...; + + SELECT create_materialized_view('mviewname','mviewname_mview', change with the name of the column to used for the index); + +The first argument is the name of the materialized view, the second the name of +the view on which the materialized view is based and the third is the column +name on which the index should be build (aka most of the time the primary key). +This column is not automatically deduced so you need to replace its name. + +As said above Ora2Pg only supports snapshot materialized views so the table will +be entirely refreshed by issuing first a truncate of the table and then by load +again all data from the view: + + refresh_full_materialized_view('mviewname'); + +To drop the materialized view you just have to call the drop_materialized_view() +function with the name of the materialized view as parameter. + +=head2 Other configuration directives + +=over 4 + +=item DEBUG + +Set it to 1 will enable verbose output. + +=item IMPORT + +You can define common Ora2Pg configuration directives into a single file that +can be imported into other configuration files with the IMPORT configuration +directive as follow: + + IMPORT commonfile.conf + +will import all configuration directives defined into commonfile.conf into the +current configuration file. + +=back + +=head2 Exporting views as PostgreSQL tables + +You can export any Oracle view as a PostgreSQL table simply by setting TYPE +configuration option to TABLE to have the corresponding create table statement. +Or use type COPY or INSERT to export the corresponding data. To allow that you +have to specify your views in the VIEW_AS_TABLE configuration option. + +Then if Ora2Pg finds the view it will extract its schema (if TYPE=TABLE) into +a PG create table form, then it will extract the data (if TYPE=COPY or INSERT) +following the view schema. + +For example, with the following view: + + CREATE OR REPLACE VIEW product_prices (category_id, product_count, low_price, high_price) AS + SELECT category_id, COUNT(*) as product_count, + MIN(list_price) as low_price, + MAX(list_price) as high_price + FROM product_information + GROUP BY category_id; + +Setting VIEW_AS_TABLE to product_prices and using export type TABLE, will +force Ora2Pg to detect columns returned types and to generate a create table +statement: + + CREATE TABLE product_prices ( + category_id bigint, + product_count integer, + low_price numeric, + high_price numeric + ); + +Data will be loaded following the COPY or INSERT export type and the view +declaration. + +You can use the ALLOW and EXCLUDE directive in addition to filter other +objects to export. + +=head2 Export as Kettle transformation XML files + +The KETTLE export type is useful if you want to use Penthalo Data Integrator +(Kettle) to import data to PostgreSQL. With this type of export Ora2Pg will +generate one XML Kettle transformation files (.ktr) per table and add a line +to manually execute the transformation in the output.sql file. For example: + + ora2pg -c ora2pg.conf -t KETTLE -j 12 -a MYTABLE -o load_mydata.sh + +will generate one file called 'HR.MYTABLE.ktr' and add a line to the output +file (load_mydata.sh): + + #!/bin/sh + + KETTLE_TEMPLATE_PATH='.' + + JAVAMAXMEM=4096 ./pan.sh -file $KETTLE_TEMPLATE_PATH/HR.MYTABLE.ktr -level Detailed + +The -j 12 option will create a template with 12 processes to insert data into +PostgreSQL. It is also possible to specify the number of parallel queries used +to extract data from the Oracle with the -J command line option as follow: + + ora2pg -c ora2pg.conf -t KETTLE -J 4 -j 12 -a EMPLOYEES -o load_mydata.sh + +This is only possible if you have defined the technical key to used to split +the query between cores in the DEFINED_PKEY configuration directive. For example: + + DEFINED_PK EMPLOYEES:employee_id + +will force the number of Oracle connection copies to 4 and defined the SQL query +as follow in the Kettle XML transformation file: + + SELECT * FROM HR.EMPLOYEES WHERE ABS(MOD(employee_id,${Internal.Step.Unique.Count}))=${Internal.Step.Unique.Number} + +The KETTLE export type requires that the Oracle and PostgreSQL DSN are defined. +You can also activate the TRUNCATE_TABLE directive to force a truncation of the +table before data import. + +The KETTLE export type is an original work of Marc Cousin. + +=head2 Migration cost assessment + +Estimating the cost of a migration process from Oracle to PostgreSQL is not easy. To +obtain a good assessment of this migration cost, Ora2Pg will inspect all database +objects, all functions and stored procedures to detect if there's still some objects +and PL/SQL code that can not be automatically converted by Ora2Pg. + +Ora2Pg has a content analysis mode that inspect the Oracle database to generate a +text report on what the Oracle database contains and what can not be exported. + +To activate the "analysis and report" mode, you have to use the export de type +SHOW_REPORT like in the following command: + + ora2pg -t SHOW_REPORT + +Here is a sample report obtained with this command: + + -------------------------------------- + Ora2Pg: Oracle Database Content Report + -------------------------------------- + Version Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 + Schema HR + Size 880.00 MB + + -------------------------------------- + Object Number Invalid Comments + -------------------------------------- + CLUSTER 2 0 Clusters are not supported and will not be exported. + FUNCTION 40 0 Total size of function code: 81992. + INDEX 435 0 232 index(es) are concerned by the export, others are automatically generated and will + do so on PostgreSQL. 1 bitmap index(es). 230 b-tree index(es). 1 reversed b-tree index(es) + Note that bitmap index(es) will be exported as b-tree index(es) if any. Cluster, domain, + bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported + too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index + and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' + operators in your indexes to improve search with the LIKE operator respectively into + varchar, text or char columns. + MATERIALIZED VIEW 1 0 All materialized view will be exported as snapshot materialized views, they + are only updated when fully refreshed. + PACKAGE BODY 2 1 Total size of package code: 20700. + PROCEDURE 7 0 Total size of procedure code: 19198. + SEQUENCE 160 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL + will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). + TABLE 265 0 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration + directive to export as file_fdw foreign tables or use COPY in your code if you just + want to load data from external files. 2 binary columns. 4 unknown types. + TABLE PARTITION 8 0 Partitions are exported using table inheritance and check constraint. 1 HASH partitions. + 2 LIST partitions. 6 RANGE partitions. Note that Hash partitions are not supported. + TRIGGER 30 0 Total size of trigger code: 21677. + TYPE 7 1 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. + 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type + inherited and Subtype are converted as table, type inheritance is not supported. + TYPE BODY 0 3 Export of type with member method are not supported, they will not be exported. + VIEW 7 0 Views are fully supported, but if you have updatable views you will need to use + INSTEAD OF triggers. + DATABASE LINK 1 0 Database links will not be exported. You may try the dblink perl contrib module or use + the SQL/MED PostgreSQL features with the different Foreign Data Wrapper (FDW) extensions. + + Note: Invalid code will not be exported unless the EXPORT_INVALID configuration directive is activated. + +Once the database can be analysed, Ora2Pg, by his ability to convert SQL and PL/SQL +code from Oracle syntax to PostgreSQL, can go further by estimating the code difficulties +and estimate the time necessary to operate a full database migration. + +To estimate the migration cost in man-days, Ora2Pg allow you to use a configuration +directive called ESTIMATE_COST that you can also enabled at command line: + + --estimate_cost + +This feature can only be used with the SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE +and QUERY export type. + + ora2pg -t SHOW_REPORT --estimate_cost + +The generated report is same as above but with a new 'Estimated cost' column as follow: + + -------------------------------------- + Ora2Pg: Oracle Database Content Report + -------------------------------------- + Version Oracle Database 10g Express Edition Release 10.2.0.1.0 + Schema HR + Size 890.00 MB + + -------------------------------------- + Object Number Invalid Estimated cost Comments + -------------------------------------- + DATABASE LINK 3 0 9 Database links will be exported as SQL/MED PostgreSQL's Foreign Data Wrapper (FDW) extensions + using oracle_fdw. + FUNCTION 2 0 7 Total size of function code: 369 bytes. HIGH_SALARY: 2, VALIDATE_SSN: 3. + INDEX 21 0 11 11 index(es) are concerned by the export, others are automatically generated and will do so + on PostgreSQL. 11 b-tree index(es). Note that bitmap index(es) will be exported as b-tree + index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. + Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a + reverse() function based index and search. You may also use 'varchar_pattern_ops', 'text_pattern_ops' + or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator + respectively into varchar, text or char columns. + JOB 0 0 0 Job are not exported. You may set external cron job with them. + MATERIALIZED VIEW 1 0 3 All materialized view will be exported as snapshot materialized views, they + are only updated when fully refreshed. + PACKAGE BODY 0 2 54 Total size of package code: 2487 bytes. Number of procedures and functions found + inside those packages: 7. two_proc.get_table: 10, emp_mgmt.create_dept: 4, + emp_mgmt.hire: 13, emp_mgmt.increase_comm: 4, emp_mgmt.increase_sal: 4, + emp_mgmt.remove_dept: 3, emp_mgmt.remove_emp: 2. + PROCEDURE 4 0 39 Total size of procedure code: 2436 bytes. TEST_COMMENTAIRE: 2, SECURE_DML: 3, + PHD_GET_TABLE: 24, ADD_JOB_HISTORY: 6. + SEQUENCE 3 0 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL + will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name'). + SYNONYM 3 0 4 SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround + is to use views or set the PostgreSQL search_path in your session to access + object outside the current schema. + user1.emp_details_view_v is an alias to hr.emp_details_view. + user1.emp_table is an alias to hr.employees@other_server. + user1.offices is an alias to hr.locations. + TABLE 17 0 8.5 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration + directive to export as file_fdw foreign tables or use COPY in your code if you just want to + load data from external files. 2 binary columns. 4 unknown types. + TRIGGER 1 1 4 Total size of trigger code: 123 bytes. UPDATE_JOB_HISTORY: 2. + TYPE 7 1 5 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. + 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are + converted as table, type inheritance is not supported. + TYPE BODY 0 3 30 Export of type with member method are not supported, they will not be exported. + VIEW 1 1 1 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. + -------------------------------------- + Total 65 8 162.5 162.5 cost migration units means approximatively 2 man day(s). + +The last line shows the total estimated migration code in man-days following the +number of migration units estimated for each object. This migration unit represent +around five minutes for a PostgreSQL expert. If this is your first migration you can +get it higher with the configuration directive COST_UNIT_VALUE or the --cost_unit_value +command line option: + + ora2pg -t SHOW_REPORT --estimate_cost --cost_unit_value 10 + +Ora2Pg is also able to give you a migration difficulty level assessment, here a sample: + +Migration level: B-5 + + Migration levels: + A - Migration that might be run automatically + B - Migration with code rewrite and a human-days cost up to 5 days + C - Migration with code rewrite and a human-days cost above 5 days + Technical levels: + 1 = trivial: no stored functions and no triggers + 2 = easy: no stored functions but with triggers, no manual rewriting + 3 = simple: stored functions and/or triggers, no manual rewriting + 4 = manual: no stored functions but with triggers or views with code rewriting + 5 = difficult: stored functions and/or triggers with code rewriting + +This assessment consist in a letter A or B to specify if the migration needs +manual rewriting or not. And a number from 1 up to 5 to give you a technical +difficulty level. You have an additional option --human_days_limit to specify +the number of human-days limit where the migration level should be set to C +to indicate that it need a huge amount of work and a full project management +with migration support. Default is 10 human-days. You can use the configuration +directive HUMAN_DAYS_LIMIT to change this default value permanently. + +This feature has been developed to help you or your boss to decide which +database to migrate first and the team that must be mobilized to operate +the migration. + +=head2 Global Oracle and MySQL migration assessment + +Ora2Pg come with a script ora2pg_scanner that can be used when you have a huge +number of instances and schema to scan for migration assessment. + +Usage: ora2pg_scanner -l CSVFILE [-o OUTDIR] + + -b | --binpath DIR: full path to directory where the ora2pg binary stays. + Might be useful only on Windows OS. + -c | --config FILE: set custom configuration file to use otherwise ora2pg + will use the default: /etc/ora2pg/ora2pg.conf. + -l | --list FILE : CSV file containing a list of databases to scan with + all required information. The first line of the file + can contain the following header that describes the + format that must be used: + + "type","schema/database","dsn","user","password" + + -o | --outdir DIR : (optional) by default all reports will be dumped to a + directory named 'output', it will be created automatically. + If you want to change the name of this directory, set the name + at second argument. + + -t | --test : just try all connections by retrieving the required schema + or database name. Useful to validate your CSV list file. + -u | --unit MIN : redefine globally the migration cost unit value in minutes. + Default is taken from the ora2pg.conf (default 5 minutes). + + Here is a full example of a CSV databases list file: + + "type","schema/database","dsn","user","password" + "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" + "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" + + The CSV field separator must be a comma. + + Note that if you want to scan all schemas from an Oracle instance you just + have to leave the schema field empty, Ora2Pg will automatically detect all + available schemas and generate a report for each one. Of course you need to + use a connection user with enough privileges to be able to scan all schemas. + For example: + + "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" + + will generate a report for all schema in the XE instance. Note that in this + case the SCHEMA directive in ora2pg.conf must not be set. + +It will generate a CSV file with the assessment result, one line per schema or +database and a detailed HTML report for each database scanned. + +Hint: Use the -t | --test option before to test all your connections in your +CSV file. + +For Windows users you must use the -b command line option to set the directory +where ora2pg_scanner stays otherwise the ora2pg command calls will fail. + +In the migration assessment details about functions Ora2Pg always include per +default 2 migration units for TEST and 1 unit for SIZE per 1000 characters in +the code. This mean that by default it will add 15 minutes in the migration +assessment per function. Obviously if you have unitary tests or very simple +functions this will not represent the real migration time. + +=head2 Migration assessment method + +Migration unit scores given to each type of Oracle database object are defined in the +Perl library lib/Ora2Pg/PLSQL.pm in the %OBJECT_SCORE variable definition. + +The number of PL/SQL lines associated to a migration unit is also defined in this file +in the $SIZE_SCORE variable value. + +The number of migration units associated to each PL/SQL code difficulties can be found +in the same Perl library lib/Ora2Pg/PLSQL.pm in the hash %UNCOVERED_SCORE initialization. + +This assessment method is a work in progress so I'm expecting feedbacks on migration +experiences to polish the scores/units attributed in those variables. + +=head2 Improving indexes and constraints creation speed + +Using the LOAD export type and a file containing SQL orders to perform, it is +possible to dispatch those orders over multiple PostgreSQL connections. To be +able to use this feature, the PG_DSN, PG_USER and PG_PWD must be set. Then: + + ora2pg -t LOAD -c config/ora2pg.conf -i schema/tables/INDEXES_table.sql -j 4 + +will dispatch indexes creation over 4 simultaneous PostgreSQL connections. + +This will considerably accelerate this part of the migration process with huge +data size. + +=head2 Exporting LONG RAW + +If you still have columns defined as LONG RAW, Ora2Pg will not be able to export +these kind of data. The OCI library fail to export them and always return the +same first record. To be able to export the data you need to transform the field +as BLOB by creating a temporary table before migrating data. For example, the +Oracle table: + + SQL> DESC TEST_LONGRAW + Name NULL ? Type + -------------------- -------- ---------------------------- + ID NUMBER + C1 LONG RAW + +need to be "translated" into a table using BLOB as follow: + + CREATE TABLE test_blob (id NUMBER, c1 BLOB); + +And then copy the data with the following INSERT query: + + INSERT INTO test_blob SELECT id, to_lob(c1) FROM test_longraw; + +Then you just have to exclude the original table from the export (see EXCLUDE +directive) and to renamed the new temporary table on the fly using the +REPLACE_TABLES configuration directive. + +=head2 Global variables + +Oracle allow the use of global variables defined in packages. Ora2Pg will +export these variables for PostgreSQL as user defined custom variables +available in a session. Oracle variables assignment are exported as +call to: + + PERFORM set_config('pkgname.varname', value, false); + +Use of these variables in the code is replaced by: + + current_setting('pkgname.varname')::global_variables_type; + +where global_variables_type is the type of the variable extracted from +the package definition. + +If the variable is a constant or have a default value assigned at +declaration, Ora2Pg will create a file global_variables.conf with +the definition to include in the postgresql.conf file so that their +values will already be set at database connection. Note that the +value can always modified by the user so you can not have exactly +a constant. + +=head2 Hints + +Converting your queries with Oracle style outer join (+) syntax to ANSI standard SQL at +the Oracle side can save you lot of time for the migration. You can use TOAD Query Builder +can re-write these using the proper ANSI syntax, see: http://www.toadworld.com/products/toad-for-oracle/f/10/t/9518.aspx + +There's also an alternative with SQL Developer Data Modeler, see +http://www.thatjeffsmith.com/archive/2012/01/sql-developer-data-modeler-quick-tip-use-oracle-join-syntax-or-ansi/ + +Toad is also able to rewrite the native Oracle DECODE() syntax into ANSI +standard SQL CASE statement. You can find some slide about this in a +presentation given at PgConf.RU: http://ora2pg.darold.net/slides/ora2pg_the_hard_way.pdf + +=head2 Test the migration + +The type of action called TEST allow you to check that all objects from Oracle +database have been created under PostgreSQL. Of course PG_DSN must be set to be +able to check PostgreSQL side. + +Note that this feature respect the schema set in the SCHEMA directive to scan +the Oracle database and also at PostgreSQL side if EXPORT_SCHEMA is enabled. +If PG_SCHEMA is defined and EXPORT_SCHEMA is enabled Ora2Pg will use the list +of schemas defined in PG_SCHEMA to scan PostgreSQL. If EXPORT_SCHEMA is +disabled the entire PostgreSQL database is scanned. + +For example command: + + ora2pg -t TEST -c config/ora2pg.conf > migration_diff.txt + +Will create a file containing the report of all object and row count on both +side, Oracle and PostgreSQL, with an error section giving you the detail of +the differences for each kind of object. Here is a sample result: + + [TEST ROWS COUNT] + ORACLEDB:COUNTRIES:25 + POSTGRES:countries:25 + ORACLEDB:CUSTOMERS:6 + POSTGRES:customers:6 + ORACLEDB:DEPARTMENTS:27 + POSTGRES:departments:27 + ORACLEDB:EMPLOYEES:107 + POSTGRES:employees:107 + ORACLEDB:JOBS:19 + POSTGRES:jobs:19 + ORACLEDB:JOB_HISTORY:10 + POSTGRES:job_history:10 + ORACLEDB:LOCATIONS:23 + POSTGRES:locations:23 + ORACLEDB:PRODUCTS:0 + POSTGRES:products:0 + ORACLEDB:PTAB2:4 + ORACLEDB:REGIONS:4 + POSTGRES:regions:4 + [ERRORS ROWS COUNT] + Table ptab2 does not exists in PostgreSQL database. + + [TEST INDEXES COUNT] + ORACLEDB:COUNTRIES:1 + POSTGRES:countries:1 + ORACLEDB:JOB_HISTORY:4 + POSTGRES:job_history:4 + ORACLEDB:DEPARTMENTS:2 + POSTGRES:departments:1 + ORACLEDB:EMPLOYEES:6 + POSTGRES:employees:6 + ORACLEDB:CUSTOMERS:1 + POSTGRES:customers:1 + ORACLEDB:REGIONS:1 + POSTGRES:regions:1 + ORACLEDB:LOCATIONS:4 + POSTGRES:locations:4 + ORACLEDB:JOBS:1 + POSTGRES:jobs:1 + [ERRORS INDEXES COUNT] + Table departments doesn't have the same number of indexes in Oracle (2) and in PostgreSQL (1). + + [TEST VIEW COUNT] + ORACLEDB:VIEW:1 + POSTGRES:VIEW:1 + [ERRORS VIEW COUNT] + OK, Oracle and PostgreSQL have the same number of VIEW. + + [TEST MVIEW COUNT] + ORACLEDB:MVIEW:0 + POSTGRES:MVIEW:0 + [ERRORS MVIEW COUNT] + OK, Oracle and PostgreSQL have the same number of MVIEW. + + [TEST SEQUENCE COUNT] + ORACLEDB:SEQUENCE:1 + POSTGRES:SEQUENCE:0 + [ERRORS SEQUENCE COUNT] + SEQUENCE does not have the same count in Oracle (1) and in PostgreSQL (0). + + [TEST TYPE COUNT] + ORACLEDB:TYPE:1 + POSTGRES:TYPE:0 + [ERRORS TYPE COUNT] + TYPE does not have the same count in Oracle (1) and in PostgreSQL (0). + + [TEST FDW COUNT] + ORACLEDB:FDW:0 + POSTGRES:FDW:0 + [ERRORS FDW COUNT] + OK, Oracle and PostgreSQL have the same number of FDW. + +Here we can see that one table, one index, one sequence and one user defined +type have not been imported yet or have encountered an error. + +=head1 SUPPORT + +=head2 Author / Maintainer + +Gilles Darold + +Please report any bugs, patches, help, etc. to . + +=head2 Feature request + +If you need new features let me know at . This help +a lot to develop a better/useful tool. + +=head2 How to contribute ? + +Any contribution to build a better tool is welcome, you just have to send me +your ideas, features request or patches and there will be applied. + +=head1 LICENSE + +Copyright (c) 2000-2020 Gilles Darold - All rights reserved. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see < http://www.gnu.org/licenses/ >. + + +=head1 ACKNOWLEDGEMENT + +I must thanks a lot all the great contributors, see changelog for all acknowledgments. + diff --git a/doc/ora2pg.3 b/doc/ora2pg.3 new file mode 100644 index 0000000000000000000000000000000000000000..fc09013811da401b40012640d3c10a3f43aad156 --- /dev/null +++ b/doc/ora2pg.3 @@ -0,0 +1,3248 @@ +.\" Automatically generated by Pod::Man 4.11 (Pod::Simple 3.35) +.\" +.\" Standard preamble: +.\" ======================================================================== +.de Sp \" Vertical space (when we can't use .PP) +.if t .sp .5v +.if n .sp +.. +.de Vb \" Begin verbatim text +.ft CW +.nf +.ne \\$1 +.. +.de Ve \" End verbatim text +.ft R +.fi +.. +.\" Set up some character translations and predefined strings. \*(-- will +.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left +.\" double quote, and \*(R" will give a right double quote. \*(C+ will +.\" give a nicer C++. Capital omega is used to do unbreakable dashes and +.\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, +.\" nothing in troff, for use with C<>. +.tr \(*W- +.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' +.ie n \{\ +. ds -- \(*W- +. ds PI pi +. if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch +. if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch +. ds L" "" +. ds R" "" +. ds C` "" +. ds C' "" +'br\} +.el\{\ +. ds -- \|\(em\| +. ds PI \(*p +. ds L" `` +. ds R" '' +. ds C` +. ds C' +'br\} +.\" +.\" Escape single quotes in literal strings from groff's Unicode transform. +.ie \n(.g .ds Aq \(aq +.el .ds Aq ' +.\" +.\" If the F register is >0, we'll generate index entries on stderr for +.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index +.\" entries marked with X<> in POD. Of course, you'll have to process the +.\" output yourself in some meaningful fashion. +.\" +.\" Avoid warning from groff about undefined register 'F'. +.de IX +.. +.nr rF 0 +.if \n(.g .if rF .nr rF 1 +.if (\n(rF:(\n(.g==0)) \{\ +. if \nF \{\ +. de IX +. tm Index:\\$1\t\\n%\t"\\$2" +.. +. if !\nF==2 \{\ +. nr % 0 +. nr F 2 +. \} +. \} +.\} +.rr rF +.\" +.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). +.\" Fear. Run. Save yourself. No user-serviceable parts. +. \" fudge factors for nroff and troff +.if n \{\ +. ds #H 0 +. ds #V .8m +. ds #F .3m +. ds #[ \f1 +. ds #] \fP +.\} +.if t \{\ +. ds #H ((1u-(\\\\n(.fu%2u))*.13m) +. ds #V .6m +. ds #F 0 +. ds #[ \& +. ds #] \& +.\} +. \" simple accents for nroff and troff +.if n \{\ +. ds ' \& +. ds ` \& +. ds ^ \& +. ds , \& +. ds ~ ~ +. ds / +.\} +.if t \{\ +. ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" +. ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' +. ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' +. ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' +. ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' +. ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' +.\} +. \" troff and (daisy-wheel) nroff accents +.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' +.ds 8 \h'\*(#H'\(*b\h'-\*(#H' +.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] +.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' +.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' +.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] +.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] +.ds ae a\h'-(\w'a'u*4/10)'e +.ds Ae A\h'-(\w'A'u*4/10)'E +. \" corrections for vroff +.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' +.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' +. \" for low resolution devices (crt and lpr) +.if \n(.H>23 .if \n(.V>19 \ +\{\ +. ds : e +. ds 8 ss +. ds o a +. ds d- d\h'-1'\(ga +. ds D- D\h'-1'\(hy +. ds th \o'bp' +. ds Th \o'LP' +. ds ae ae +. ds Ae AE +.\} +.rm #[ #] #H #V #F C +.\" ======================================================================== +.\" +.IX Title "ORA2PG 1" +.TH ORA2PG 1 "2021-04-01" "perl v5.30.0" "User Contributed Perl Documentation" +.\" For nroff, turn off justification. Always turn off hyphenation; it makes +.\" way too many mistakes in technical documents. +.if n .ad l +.nh +.SH "NAME" +Ora2Pg \- Oracle to PostgreSQL database schema converter +.SH "DESCRIPTION" +.IX Header "DESCRIPTION" +Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL +compatible schema. It connects your Oracle database, scans it automatically +and extracts its structure or data, then generates \s-1SQL\s0 scripts that you can +load into your PostgreSQL database. +.PP +Ora2Pg can be used for anything from reverse engineering Oracle database to +huge enterprise database migration or simply replicating some Oracle data into +a PostgreSQL database. It is really easy to use and doesn't require any Oracle +database knowledge other than providing the parameters needed to connect to the +Oracle database. +.SH "FEATURES" +.IX Header "FEATURES" +Ora2Pg consist of a Perl script (ora2pg) and a Perl module (Ora2Pg.pm), the +only thing you have to modify is the configuration file ora2pg.conf by setting +the \s-1DSN\s0 to the Oracle database and optionally the name of a schema. Once that's +done you just have to set the type of export you want: \s-1TABLE\s0 with constraints, +\&\s-1VIEW, MVIEW, TABLESPACE, SEQUENCE, INDEXES, TRIGGER, GRANT, FUNCTION, PROCEDURE, +PACKAGE, PARTITION, TYPE, INSERT\s0 or \s-1COPY, FDW, QUERY, KETTLE, SYNONYM.\s0 +.PP +By default Ora2Pg exports to a file that you can load into PostgreSQL with the +psql client, but you can also import directly into a PostgreSQL database by +setting its \s-1DSN\s0 into the configuration file. With all configuration options of +ora2pg.conf you have full control of what should be exported and how. +.PP +Features included: +.PP +.Vb 10 +\& \- Export full database schema (tables, views, sequences, indexes), with +\& unique, primary, foreign key and check constraints. +\& \- Export grants/privileges for users and groups. +\& \- Export range/list partitions and sub partitions. +\& \- Export a table selection (by specifying the table names). +\& \- Export Oracle schema to a PostgreSQL 8.4+ schema. +\& \- Export predefined functions, triggers, procedures, packages and +\& package bodies. +\& \- Export full data or following a WHERE clause. +\& \- Full support of Oracle BLOB object as PG BYTEA. +\& \- Export Oracle views as PG tables. +\& \- Export Oracle user defined types. +\& \- Provide some basic automatic conversion of PLSQL code to PLPGSQL. +\& \- Works on any platform. +\& \- Export Oracle tables as foreign data wrapper tables. +\& \- Export materialized view. +\& \- Show a report of an Oracle database content. +\& \- Migration cost assessment of an Oracle database. +\& \- Migration difficulty level assessment of an Oracle database. +\& \- Migration cost assessment of PL/SQL code from a file. +\& \- Migration cost assessment of Oracle SQL queries stored in a file. +\& \- Generate XML ktr files to be used with Penthalo Data Integrator (Kettle) +\& \- Export Oracle locator and spatial geometries into PostGis. +\& \- Export DBLINK as Oracle FDW. +\& \- Export SYNONYMS as views. +\& \- Export DIRECTORY as external table or directory for external_file extension. +\& \- Full MySQL export just like Oracle database. +\& \- Dispatch a list of SQL orders over multiple PostgreSQL connections +\& \- Perform a diff between Oracle and PostgreSQL database for test purpose. +.Ve +.PP +Ora2Pg does its best to automatically convert your Oracle database to PostgreSQL +but there's still manual works to do. The Oracle specific \s-1PL/SQL\s0 code generated +for functions, procedures, packages and triggers has to be reviewed to match +the PostgreSQL syntax. You will find some useful recommendations on porting +Oracle \s-1PL/SQL\s0 code to PostgreSQL \s-1PL/PGSQL\s0 at \*(L"Converting from other Databases +to PostgreSQL\*(R", section: Oracle (http://wiki.postgresql.org/wiki/Main_Page). +.PP +See http://ora2pg.darold.net/report.html for a \s-1HTML\s0 sample of an Oracle database +migration report. +.SH "INSTALLATION" +.IX Header "INSTALLATION" +All Perl modules can always be found at \s-1CPAN\s0 (http://search.cpan.org/). Just +type the full name of the module (ex: DBD::Oracle) into the search input box, +it will brings you the page for download. +.PP +Releases of Ora2Pg stay at \s-1SF\s0.net (https://sourceforge.net/projects/ora2pg/). +.PP +Under Windows you should install Strawberry Perl (http://strawberryperl.com/) +and the OSes corresponding Oracle clients. Since version 5.32 this Perl +distribution include pre-compiled driver of DBD::Oracle and DBD::Pg. +.SS "Requirement" +.IX Subsection "Requirement" +The Oracle Instant Client or a full Oracle installation must be installed on +the system. You can download the \s-1RPM\s0 from Oracle download center: +.PP +.Vb 4 +\& rpm \-ivh oracle\-instantclient12.2\-basic\-12.2.0.1.0\-1.x86_64.rpm +\& rpm \-ivh oracle\-instantclient12.2\-devel\-12.2.0.1.0\-1.x86_64.rpm +\& rpm \-ivh oracle\-instantclient12.2\-jdbc\-12.2.0.1.0\-1.x86_64.rpm +\& rpm \-ivh oracle\-instantclient12.2\-sqlplus\-12.2.0.1.0\-1.x86_64.rpm +.Ve +.PP +or simply download the corresponding \s-1ZIP\s0 archives from Oracle download center +and install them where you want, for example: /opt/oracle/instantclient_12_2/ +.PP +You also need a modern Perl distribution (perl 5.10 and more). To connect to a +database and proceed to his migration you need the \s-1DBI\s0 Perl module > 1.614. +To migrate an Oracle database you need the DBD::Oracle Perl modules to be +installed. To migrate a MySQL database you need the DBD::MySQL Perl modules. +These modules are used to connect to the database but they are not mandatory +if you want to migrate \s-1DDL\s0 input files. +.PP +To install DBD::Oracle and have it working you need to have the Oracle client +libraries installed and the \s-1ORACLE_HOME\s0 environment variable must be defined. +.PP +If you plan to export a MySQL database you need to install the Perl module +DBD::mysql which requires that the mysql client libraries are installed. +.PP +On some Perl distribution you may need to install the Time::HiRes Perl module. +.PP +If your distribution doesn't include these Perl modules you can install them +using \s-1CPAN:\s0 +.PP +.Vb 3 +\& perl \-MCPAN \-e \*(Aqinstall DBD::Oracle\*(Aq +\& perl \-MCPAN \-e \*(Aqinstall DBD::MySQL\*(Aq +\& perl \-MCPAN \-e \*(Aqinstall Time::HiRes\*(Aq +.Ve +.PP +otherwise use the packages provided by your distribution. +.SS "Optional" +.IX Subsection "Optional" +By default Ora2Pg dumps export to flat files, to load them into your PostgreSQL +database you need the PostgreSQL client (psql). If you don't have it on the +host running Ora2Pg you can always transfer these files to a host with the psql +client installed. If you prefer to load export 'on the fly', the perl module +DBD::Pg is required. +.PP +Ora2Pg allows you to dump all output in a compressed gzip file, to do that you +need the Compress::Zlib Perl module or if you prefer using bzip2 compression, +the program bzip2 must be available in your \s-1PATH.\s0 +.PP +If your distribution doesn't include these Perl modules you can install them +using \s-1CPAN:\s0 +.PP +.Vb 2 +\& perl \-MCPAN \-e \*(Aqinstall DBD::Pg\*(Aq +\& perl \-MCPAN \-e \*(Aqinstall Compress::Zlib\*(Aq +.Ve +.PP +otherwise use the packages provided by your distribution. +.SS "Installing Ora2Pg" +.IX Subsection "Installing Ora2Pg" +Like any other Perl Module Ora2Pg can be installed with the following commands: +.PP +.Vb 4 +\& tar xjf ora2pg\-x.x.tar.bz2 +\& cd ora2pg\-x.x/ +\& perl Makefile.PL +\& make && make install +.Ve +.PP +This will install Ora2Pg.pm into your site Perl repository, ora2pg into +/usr/local/bin/ and ora2pg.conf into /etc/ora2pg/. +.PP +On Windows(tm) OSes you may use instead: +.PP +.Vb 2 +\& perl Makefile.PL +\& dmake && dmake install +.Ve +.PP +This will install scripts and libraries into your Perl site installation +directory and the ora2pg.conf file as well as all documentation files +into C:\eora2pg\e +.PP +To install ora2pg in a different directory than the default one, simply +use this command: +.PP +.Vb 2 +\& perl Makefile.PL PREFIX= +\& make && make install +.Ve +.PP +then set \s-1PERL5LIB\s0 to the path to your installation directory before using +Ora2Pg. +.PP +.Vb 2 +\& export PERL5LIB= +\& ora2pg \-c config/ora2pg.conf \-t TABLE \-b outdir/ +.Ve +.SS "Packaging" +.IX Subsection "Packaging" +If you want to build the binary package for your preferred Linux distribution +take a look at the packaging/ directory of the source tarball. There is +everything to build \s-1RPM,\s0 Slackware and Debian packages. See \s-1README\s0 file in +that directory. +.SS "Installing DBD::Oracle" +.IX Subsection "Installing DBD::Oracle" +Ora2Pg needs the Perl module DBD::Oracle for connectivity to an Oracle database +from perl \s-1DBI.\s0 To get DBD::Oracle get it from \s-1CPAN\s0 a perl module repository. +.PP +After setting \s-1ORACLE_HOME\s0 and \s-1LD_LIBRARY_PATH\s0 environment variables as root +user, install DBD::Oracle. Proceed as follow: +.PP +.Vb 3 +\& export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib +\& export ORACLE_HOME=/usr/lib/oracle/12.2/client64 +\& perl \-MCPAN \-e \*(Aqinstall DBD::Oracle\*(Aq +.Ve +.PP +If you are running for the first time it will ask many questions; you can keep +defaults by pressing \s-1ENTER\s0 key, but you need to give one appropriate mirror +site for \s-1CPAN\s0 to download the modules. Install through \s-1CPAN\s0 manually if the +above doesn't work: +.PP +.Vb 9 +\& #perl \-MCPAN \-e shell +\& cpan> get DBD::Oracle +\& cpan> quit +\& cd ~/.cpan/build/DBD\-Oracle* +\& export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib +\& export ORACLE_HOME=/usr/lib/oracle/11.2/client64 +\& perl Makefile.PL +\& make +\& make install +.Ve +.PP +Installing DBD::Oracle require that the three Oracle packages: instant-client, +\&\s-1SDK\s0 and SQLplus are installed as well as the libaio1 library. +.PP +If you are using Instant Client from \s-1ZIP\s0 archives, the \s-1LD_LIBRARY_PATH\s0 and +\&\s-1ORACLE_HOME\s0 will be the same and must be set to the directory where you have +installed the files. For example: /opt/oracle/instantclient_12_2/ +.SH "CONFIGURATION" +.IX Header "CONFIGURATION" +Ora2Pg configuration can be as simple as choosing the Oracle database to export +and choose the export type. This can be done in a minute. +.PP +By reading this documentation you will also be able to: +.PP +.Vb 6 +\& \- Select only certain tables and/or column for export. +\& \- Rename some tables and/or column during export. +\& \- Select data to export following a WHERE clause per table. +\& \- Delay database constraints during data loading. +\& \- Compress exported data to save disk space. +\& \- and much more. +.Ve +.PP +The full control of the Oracle database migration is taken though a single +configuration file named ora2pg.conf. The format of this file consist in a +directive name in upper case followed by tab character and a value. +Comments are lines beginning with a #. +.PP +There's no specific order to place the configuration directives, they are +set at the time they are read in the configuration file. +.PP +For configuration directives that just take a single value, you can use them +multiple time in the configuration file but only the last occurrence found +in the file will be used. For configuration directives that allow a list +of value, you can use it multiple time, the values will be appended to the +list. If you use the \s-1IMPORT\s0 directive to load a custom configuration file, +directives defined in this file will be stores from the place the \s-1IMPORT\s0 +directive is found, so it is better to put it at the end of the configuration +file. +.PP +Values set in command line options will override values from the configuration +file. +.SS "Ora2Pg usage" +.IX Subsection "Ora2Pg usage" +First of all be sure that libraries and binaries path include the Oracle +Instant Client installation: +.PP +.Vb 2 +\& export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib +\& export PATH="/usr/lib/oracle/11.2/client64/bin:$PATH" +.Ve +.PP +By default Ora2Pg will look for /etc/ora2pg/ora2pg.conf configuration file, if +the file exist you can simply execute: +.PP +.Vb 1 +\& /usr/local/bin/ora2pg +.Ve +.PP +or under Windows(tm) run ora2pg.bat file, located in your perl bin directory. +Windows(tm) users may also find a template configuration file in C:\eora2pg +.PP +If you want to call another configuration file, just give the path as command +line argument: +.PP +.Vb 1 +\& /usr/local/bin/ora2pg \-c /etc/ora2pg/new_ora2pg.conf +.Ve +.PP +Here are all command line parameters available when using ora2pg: +.PP +Usage: ora2pg [\-dhpqv \-\-estimate_cost \-\-dump_as_html] [\-\-option value] +.PP +.Vb 10 +\& \-a | \-\-allow str : Comma separated list of objects to allow from export. +\& Can be used with SHOW_COLUMN too. +\& \-b | \-\-basedir dir: Set the default output directory, where files +\& resulting from exports will be stored. +\& \-c | \-\-conf file : Set an alternate configuration file other than the +\& default /etc/ora2pg/ora2pg.conf. +\& \-d | \-\-debug : Enable verbose output. +\& \-D | \-\-data_type STR : Allow custom type replacement at command line. +\& \-e | \-\-exclude str: Comma separated list of objects to exclude from export. +\& Can be used with SHOW_COLUMN too. +\& \-h | \-\-help : Print this short help. +\& \-g | \-\-grant_object type : Extract privilege from the given object type. +\& See possible values with GRANT_OBJECT configuration. +\& \-i | \-\-input file : File containing Oracle PL/SQL code to convert with +\& no Oracle database connection initiated. +\& \-j | \-\-jobs num : Number of parallel process to send data to PostgreSQL. +\& \-J | \-\-copies num : Number of parallel connections to extract data from Oracle. +\& \-l | \-\-log file : Set a log file. Default is stdout. +\& \-L | \-\-limit num : Number of tuples extracted from Oracle and stored in +\& memory before writing, default: 10000. +\& \-m | \-\-mysql : Export a MySQL database instead of an Oracle schema. +\& \-n | \-\-namespace schema : Set the Oracle schema to extract from. +\& \-N | \-\-pg_schema schema : Set PostgreSQL\*(Aqs search_path. +\& \-o | \-\-out file : Set the path to the output file where SQL will +\& be written. Default: output.sql in running directory. +\& \-p | \-\-plsql : Enable PLSQL to PLPGSQL code conversion. +\& \-P | \-\-parallel num: Number of parallel tables to extract at the same time. +\& \-q | \-\-quiet : Disable progress bar. +\& \-r | \-\-relative : use \eir instead of \ei in the psql scripts generated. +\& \-s | \-\-source DSN : Allow to set the Oracle DBI datasource. +\& \-t | \-\-type export: Set the export type. It will override the one +\& given in the configuration file (TYPE). +\& \-T | \-\-temp_dir DIR: Set a distinct temporary directory when two +\& or more ora2pg are run in parallel. +\& \-u | \-\-user name : Set the Oracle database connection user. +\& ORA2PG_USER environment variable can be used instead. +\& \-v | \-\-version : Show Ora2Pg Version and exit. +\& \-w | \-\-password pwd : Set the password of the Oracle database user. +\& ORA2PG_PASSWD environment variable can be used instead. +\& \-\-forceowner : Force ora2pg to set tables and sequences owner like in +\& Oracle database. If the value is set to a username this one +\& will be used as the objects owner. By default it\*(Aqs the user +\& used to connect to the Pg database that will be the owner. +\& \-\-nls_lang code: Set the Oracle NLS_LANG client encoding. +\& \-\-client_encoding code: Set the PostgreSQL client encoding. +\& \-\-view_as_table str: Comma separated list of views to export as table. +\& \-\-estimate_cost : Activate the migration cost evaluation with SHOW_REPORT +\& \-\-cost_unit_value minutes: Number of minutes for a cost evaluation unit. +\& default: 5 minutes, corresponds to a migration conducted by a +\& PostgreSQL expert. Set it to 10 if this is your first migration. +\& \-\-dump_as_html : Force ora2pg to dump report in HTML, used only with +\& SHOW_REPORT. Default is to dump report as simple text. +\& \-\-dump_as_csv : As above but force ora2pg to dump report in CSV. +\& \-\-dump_as_sheet : Report migration assessment with one CSV line per database. +\& \-\-init_project NAME: Initialise a typical ora2pg project tree. Top directory +\& will be created under project base dir. +\& \-\-project_base DIR : Define the base dir for ora2pg project trees. Default +\& is current directory. +\& \-\-print_header : Used with \-\-dump_as_sheet to print the CSV header +\& especially for the first run of ora2pg. +\& \-\-human_days_limit num : Set the number of human\-days limit where the migration +\& assessment level switch from B to C. Default is set to +\& 5 human\-days. +\& \-\-audit_user LIST : Comma separated list of usernames to filter queries in +\& the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT +\& and QUERY export type. +\& \-\-pg_dsn DSN : Set the datasource to PostgreSQL for direct import. +\& \-\-pg_user name : Set the PostgreSQL user to use. +\& \-\-pg_pwd password : Set the PostgreSQL password to use. +\& \-\-count_rows : Force ora2pg to perform a real row count in TEST action. +\& \-\-no_header : Do not append Ora2Pg header to output file +\& \-\-oracle_speed : Use to know at which speed Oracle is able to send +\& data. No data will be processed or written. +\& \-\-ora2pg_speed : Use to know at which speed Ora2Pg is able to send +\& transformed data. Nothing will be written. +.Ve +.PP +See full documentation at http://ora2pg.darold.net/ for more help or see +manpage with 'man ora2pg'. +.PP +ora2pg will return 0 on success, 1 on error. It will return 2 when a child +process has been interrupted and you've gotten the warning message: + \*(L"\s-1WARNING:\s0 an error occurs during data export. Please check what's happen.\*(R" +Most of the time this is an \s-1OOM\s0 issue, first try reducing \s-1DATA_LIMIT\s0 value. +.PP +For developers, it is possible to add your own custom option(s) in the Perl +script ora2pg as any configuration directive from ora2pg.conf can be passed +in lower case to the new Ora2Pg object instance. See ora2pg code on how to +add your own option. +.PP +Note that performance might be improved by updating stats on oracle: +.PP +.Vb 5 +\& BEGIN +\& DBMS_STATS.GATHER_SCHEMA_STATS +\& DBMS_STATS.GATHER_DATABASE_STATS +\& DBMS_STATS.GATHER_DICTIONARY_STATS +\& END; +.Ve +.SS "Generate a migration template" +.IX Subsection "Generate a migration template" +The two options \-\-project_base and \-\-init_project when used indicate to ora2pg +that he has to create a project template with a work tree, a configuration +file and a script to export all objects from the Oracle database. Here a sample +of the command usage: +.PP +.Vb 10 +\& ora2pg \-\-project_base /app/migration/ \-\-init_project test_project +\& Creating project test_project. +\& /app/migration/test_project/ +\& schema/ +\& dblinks/ +\& directories/ +\& functions/ +\& grants/ +\& mviews/ +\& packages/ +\& partitions/ +\& procedures/ +\& sequences/ +\& synonyms/ +\& tables/ +\& tablespaces/ +\& triggers/ +\& types/ +\& views/ +\& sources/ +\& functions/ +\& mviews/ +\& packages/ +\& partitions/ +\& procedures/ +\& triggers/ +\& types/ +\& views/ +\& data/ +\& config/ +\& reports/ +\& +\& Generating generic configuration file +\& Creating script export_schema.sh to automate all exports. +\& Creating script import_all.sh to automate all imports. +.Ve +.PP +It create a generic config file where you just have to define the Oracle +database connection and a shell script called export_schema.sh. The sources/ +directory will contains the Oracle code, the schema/ will contains the code +ported to PostgreSQL. The reports/ directory will contains the html reports +with the migration cost assessment. +.PP +If you want to use your own default config file, use the \-c option to give +the path to that file. Rename it with .dist suffix if you want ora2pg to +apply the generic configuration values otherwise, the configuration file +will be copied untouched. +.PP +Once you have set the connection to the Oracle Database you can execute the +script export_schema.sh that will export all object type from your Oracle +database and output \s-1DDL\s0 files into the schema's subdirectories. At end of the +export it will give you the command to export data later when the import of +the schema will be done and verified. +.PP +You can choose to load the \s-1DDL\s0 files generated manually or use the second +script import_all.sh to import those file interactively. If this kind of +migration is not something current for you it's recommended you to use those +scripts. +.SS "Oracle database connection" +.IX Subsection "Oracle database connection" +There's 5 configuration directives to control the access to the Oracle database. +.IP "\s-1ORACLE_HOME\s0" 4 +.IX Item "ORACLE_HOME" +Used to set \s-1ORACLE_HOME\s0 environment variable to the Oracle libraries required +by the DBD::Oracle Perl module. +.IP "\s-1ORACLE_DSN\s0" 4 +.IX Item "ORACLE_DSN" +This directive is used to set the data source name in the form standard \s-1DBI DSN.\s0 +For example: +.Sp +.Vb 1 +\& dbi:Oracle:host=oradb_host.myhost.com;sid=DB_SID;port=1521 +.Ve +.Sp +or +.Sp +.Vb 1 +\& dbi:Oracle:DB_SID +.Ve +.Sp +On 18c this could be for example: +.Sp +.Vb 1 +\& dbi:Oracle:host=192.168.1.29;service_name=pdb1;port=1521 +.Ve +.Sp +for the second notation the \s-1SID\s0 should be declared in the well known file +\&\f(CW$ORACLE_HOME\fR/network/admin/tnsnames.ora or in the path given to the \s-1TNS_ADMIN\s0 +environment variable. +.Sp +For MySQL the \s-1DSN\s0 will lool like this: +.Sp +.Vb 1 +\& dbi:mysql:host=192.168.1.10;database=sakila;port=3306 +.Ve +.Sp +the 'sid' part is replaced by 'database'. +.IP "\s-1ORACLE_USER\s0 et \s-1ORACLE_PWD\s0" 4 +.IX Item "ORACLE_USER et ORACLE_PWD" +These two directives are used to define the user and password for the Oracle +database connection. Note that if you can it is better to login as Oracle super +admin to avoid grants problem during the database scan and be sure that nothing +is missing. +.Sp +If you do not supply a credential with \s-1ORACLE_PWD\s0 and you have installed the +Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If +\&\s-1ORACLE_USER\s0 is not set it will be asked interactively too. +.Sp +To connect to a local \s-1ORACLE\s0 instance with connections \*(L"as sysdba\*(R" you have to +set \s-1ORACLE_USER\s0 to \*(L"/\*(R" and an empty password. +.IP "\s-1USER_GRANTS\s0" 4 +.IX Item "USER_GRANTS" +Set this directive to 1 if you connect the Oracle database as simple user and +do not have enough grants to extract things from the \s-1DBA_...\s0 tables. It will +use tables \s-1ALL_...\s0 instead. +.Sp +Warning: if you use export type \s-1GRANT,\s0 you must set this configuration option +to 0 or it will not work. +.IP "\s-1TRANSACTION\s0" 4 +.IX Item "TRANSACTION" +This directive may be used if you want to change the default isolation level of +the data export transaction. Default is now to set the level to a serializable +transaction to ensure data consistency. The allowed values for this directive +are: +.Sp +.Vb 4 +\& readonly: \*(AqSET TRANSACTION READ ONLY\*(Aq, +\& readwrite: \*(AqSET TRANSACTION READ WRITE\*(Aq, +\& serializable: \*(AqSET TRANSACTION ISOLATION LEVEL SERIALIZABLE\*(Aq +\& committed: \*(AqSET TRANSACTION ISOLATION LEVEL READ COMMITTED\*(Aq, +.Ve +.Sp +Releases before 6.2 used to set the isolation level to \s-1READ ONLY\s0 transaction +but in some case this was breaking data consistency so now default is set to +\&\s-1SERIALIZABLE.\s0 +.IP "\s-1INPUT_FILE\s0" 4 +.IX Item "INPUT_FILE" +This directive did not control the Oracle database connection or unless it +purely disables the use of any Oracle database by accepting a file as argument. +Set this directive to a file containing \s-1PL/SQL\s0 Oracle Code like function, +procedure or full package body to prevent Ora2Pg from connecting to an +Oracle database and just apply his conversion tool to the content of the +file. This can be used with the most of export types: \s-1TABLE, TRIGGER, PROCEDURE, +VIEW, FUNCTION\s0 or \s-1PACKAGE,\s0 etc. +.IP "\s-1ORA_INITIAL_COMMAND\s0" 4 +.IX Item "ORA_INITIAL_COMMAND" +This directive can be used to send an initial command to Oracle, just after +the connection. For example to unlock a policy before reading objects or +to set some session parameters. This directive can be used multiple times. +.SS "Data encryption with Oracle server" +.IX Subsection "Data encryption with Oracle server" +If your Oracle Client config file already includes the encryption method, +then DBD:Oracle uses those settings to encrypt the connection while you +extract the data. For example if you have configured the Oracle Client +config file (sqlnet.or or .sqlnet) with the following information: +.PP +.Vb 4 +\& # Configure encryption of connections to Oracle +\& SQLNET.ENCRYPTION_CLIENT = required +\& SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256, RC4_256) +\& SQLNET.CRYPTO_SEED = \*(Aqshould be 10\-70 random characters\*(Aq +.Ve +.PP +Any tool that uses the Oracle client to talk to the database will be +encrypted if you setup session encryption like above. +.PP +For example, Perl's \s-1DBI\s0 uses DBD-Oracle, which uses the Oracle client +for actually handling database communication. If the installation of +Oracle client used by Perl is setup to request encrypted connections, +then your Perl connection to an Oracle database will also be encrypted. +.PP +Full details at https://kb.berkeley.edu/jivekb/entry.jspa?externalID=1005 +.SS "Testing connection" +.IX Subsection "Testing connection" +Once you have set the Oracle database \s-1DSN\s0 you can execute ora2pg to see if +it works: +.PP +.Vb 1 +\& ora2pg \-t SHOW_VERSION \-c config/ora2pg.conf +.Ve +.PP +will show the Oracle database server version. Take some time here to test your +installation as most problems take place here, the other configuration +steps are more technical. +.SS "Troubleshooting" +.IX Subsection "Troubleshooting" +If the output.sql file has not exported anything other than the Pg transaction +header and footer there's two possible reasons. The perl script ora2pg dump +an ORA-XXX error, that mean that your \s-1DSN\s0 or login information are wrong, check +the error and your settings and try again. The perl script says nothing and the +output file is empty: the user lacks permission to extract something from +the database. Try to connect to Oracle as super user or take a look at directive +\&\s-1USER_GRANTS\s0 above and at next section, especially the \s-1SCHEMA\s0 directive. +.IP "\s-1LOGFILE\s0" 4 +.IX Item "LOGFILE" +By default all messages are sent to the standard output. If you give a file +path to that directive, all output will be appended to this file. +.SS "Oracle schema to export" +.IX Subsection "Oracle schema to export" +The Oracle database export can be limited to a specific Schema or Namespace, +this can be mandatory following the database connection user. +.IP "\s-1SCHEMA\s0" 4 +.IX Item "SCHEMA" +This directive is used to set the schema name to use during export. +For example: +.Sp +.Vb 1 +\& SCHEMA APPS +.Ve +.Sp +will extract objects associated to the \s-1APPS\s0 schema. +.Sp +When no schema name is provided and \s-1EXPORT_SCHEMA\s0 is enabled, Ora2Pg +will export all objects from all schema of the Oracle instance with +their names prefixed with the schema name. +.IP "\s-1EXPORT_SCHEMA\s0" 4 +.IX Item "EXPORT_SCHEMA" +By default the Oracle schema is not exported into the PostgreSQL database and +all objects are created under the default Pg namespace. If you want to also +export this schema and create all objects under this namespace, set the +\&\s-1EXPORT_SCHEMA\s0 directive to 1. This will set the schema search_path at top of +export \s-1SQL\s0 file to the schema name set in the \s-1SCHEMA\s0 directive with the default +pg_catalog schema. If you want to change this path, use the directive \s-1PG_SCHEMA.\s0 +.IP "\s-1CREATE_SCHEMA\s0" 4 +.IX Item "CREATE_SCHEMA" +Enable/disable the \s-1CREATE SCHEMA SQL\s0 order at starting of the output file. +It is enable by default and concern on \s-1TABLE\s0 export type. +.IP "\s-1COMPILE_SCHEMA\s0" 4 +.IX Item "COMPILE_SCHEMA" +By default Ora2Pg will only export valid \s-1PL/SQL\s0 code. You can force Oracle to +compile again the invalidated code to get a chance to have it obtain the valid +status and then be able to export it. +.Sp +Enable this directive to force Oracle to compile schema before exporting code. +When this directive is enabled and \s-1SCHEMA\s0 is set to a specific schema name, +only invalid objects in this schema will be recompiled. If \s-1SCHEMA\s0 is not set +then all schema will be recompiled. To force recompile invalid object in a +specific schema, set \s-1COMPILE_SCHEMA\s0 to the schema name you want to recompile. +.Sp +This will ask to Oracle to validate the \s-1PL/SQL\s0 that could have been invalidate +after a export/import for example. The '\s-1VALID\s0' or '\s-1INVALID\s0' status applies to +functions, procedures, packages and user defined types. +.IP "\s-1EXPORT_INVALID\s0" 4 +.IX Item "EXPORT_INVALID" +If the above configuration directive is not enough to validate your \s-1PL/SQL\s0 code +enable this configuration directive to allow export of all \s-1PL/SQL\s0 code even if +it is marked as invalid. The '\s-1VALID\s0' or '\s-1INVALID\s0' status applies to functions, +procedures, packages and user defined types. +.IP "\s-1PG_SCHEMA\s0" 4 +.IX Item "PG_SCHEMA" +Allow you to defined/force the PostgreSQL schema to use. By default if you set +\&\s-1EXPORT_SCHEMA\s0 to 1 the PostgreSQL search_path will be set to the schema name +exported set as value of the \s-1SCHEMA\s0 directive. +.Sp +The value can be a comma delimited list of schema name but not when using \s-1TABLE\s0 +export type because in this case it will generate the \s-1CREATE SCHEMA\s0 statement +and it doesn't support multiple schema name. For example, if you set \s-1PG_SCHEMA\s0 +to something like \*(L"user_schema, public\*(R", the search path will be set like this: +.Sp +.Vb 1 +\& SET search_path = user_schema, public; +.Ve +.Sp +forcing the use of an other schema (here user_schema) than the one from Oracle +schema set in the \s-1SCHEMA\s0 directive. +.Sp +You can also set the default search_path for the PostgreSQL user you are using +to connect to the destination database by using: +.Sp +.Vb 1 +\& ALTER ROLE username SET search_path TO user_schema, public; +.Ve +.Sp +in this case you don't have to set \s-1PG_SCHEMA.\s0 +.IP "\s-1SYSUSERS\s0" 4 +.IX Item "SYSUSERS" +Without explicit schema, Ora2Pg will export all objects that not belongs to +system schema or role: +.Sp +.Vb 12 +\& SYSTEM,CTXSYS,DBSNMP,EXFSYS,LBACSYS,MDSYS,MGMT_VIEW, +\& OLAPSYS,ORDDATA,OWBSYS,ORDPLUGINS,ORDSYS,OUTLN, +\& SI_INFORMTN_SCHEMA,SYS,SYSMAN,WK_TEST,WKSYS,WKPROXY, +\& WMSYS,XDB,APEX_PUBLIC_USER,DIP,FLOWS_020100,FLOWS_030000, +\& FLOWS_040100,FLOWS_010600,FLOWS_FILES,MDDATA,ORACLE_OCM, +\& SPATIAL_CSW_ADMIN_USR,SPATIAL_WFS_ADMIN_USR,XS$NULL,PERFSTAT, +\& SQLTXPLAIN,DMSYS,TSMSYS,WKSYS,APEX_040000,APEX_040200, +\& DVSYS,OJVMSYS,GSMADMIN_INTERNAL,APPQOSSYS,DVSYS,DVF, +\& AUDSYS,APEX_030200,MGMT_VIEW,ODM,ODM_MTR,TRACESRV,MTMSYS, +\& OWBSYS_AUDIT,WEBSYS,WK_PROXY,OSE$HTTP$ADMIN, +\& AURORA$JIS$UTILITY$,AURORA$ORB$UNAUTHENTICATED, +\& DBMS_PRIVILEGE_CAPTURE,CSMIG,MGDSYS,SDE,DBSFWUSER +.Ve +.Sp +Following your Oracle installation you may have several other system role +defined. To append these users to the schema exclusion list, just set the +\&\s-1SYSUSERS\s0 configuration directive to a comma-separated list of system user to +exclude. For example: +.Sp +.Vb 1 +\& SYSUSERS INTERNAL,SYSDBA,BI,HR,IX,OE,PM,SH +.Ve +.Sp +will add users \s-1INTERNAL\s0 and \s-1SYSDBA\s0 to the schema exclusion list. +.IP "\s-1FORCE_OWNER\s0" 4 +.IX Item "FORCE_OWNER" +By default the owner of the database objects is the one you're using to connect +to PostgreSQL using the psql command. If you use an other user (postgres for example) +you can force Ora2Pg to set the object owner to be the one used in the Oracle database +by setting the directive to 1, or to a completely different username by setting the +directive value to that username. +.IP "\s-1FORCE_SECURITY_INVOKER\s0" 4 +.IX Item "FORCE_SECURITY_INVOKER" +Ora2Pg use the function's security privileges set in Oracle and it is often +defined as \s-1SECURITY DEFINER.\s0 If you want to override those security privileges +for all functions and use \s-1SECURITY DEFINER\s0 instead, enable this directive. +.IP "\s-1USE_TABLESPACE\s0" 4 +.IX Item "USE_TABLESPACE" +When enabled this directive force ora2pg to export all tables, indexes constraint and +indexes using the tablespace name defined in Oracle database. This works only with +tablespace that are not \s-1TEMP, USERS\s0 and \s-1SYSTEM.\s0 +.IP "\s-1WITH_OID\s0" 4 +.IX Item "WITH_OID" +Activating this directive will force Ora2Pg to add \s-1WITH\s0 (\s-1OIDS\s0) when creating +tables or views as tables. Default is same as PostgreSQL, disabled. +.IP "\s-1LOOK_FORWARD_FUNCTION\s0" 4 +.IX Item "LOOK_FORWARD_FUNCTION" +List of schema to get functions/procedures meta information that are used +in the current schema export. When replacing call to function with \s-1OUT\s0 +parameters, if a function is declared in an other package then the function +call rewriting can not be done because Ora2Pg only knows about functions +declared in the current schema. By setting a comma separated list of schema +as value of this directive, Ora2Pg will look forward in these packages for +all functions/procedures/packages declaration before proceeding to current +schema export. +.IP "\s-1NO_FUNCTION_METADATA\s0" 4 +.IX Item "NO_FUNCTION_METADATA" +Force Ora2Pg to not look for function declaration. Note that this will prevent +Ora2Pg to rewrite function replacement call if needed. Do not enable it unless +looking forward at function breaks other export. +.SS "Export type" +.IX Subsection "Export type" +The export action is perform following a single configuration directive '\s-1TYPE\s0', +some other add more control on what should be really exported. +.IP "\s-1TYPE\s0" 4 +.IX Item "TYPE" +Here are the different values of the \s-1TYPE\s0 directive, default is \s-1TABLE:\s0 +.Sp +.Vb 10 +\& \- TABLE: Extract all tables with indexes, primary keys, unique keys, +\& foreign keys and check constraints. +\& \- VIEW: Extract only views. +\& \- GRANT: Extract roles converted to Pg groups, users and grants on all +\& objects. +\& \- SEQUENCE: Extract all sequence and their last position. +\& \- TABLESPACE: Extract storage spaces for tables and indexes (Pg >= v8). +\& \- TRIGGER: Extract triggers defined following actions. +\& \- FUNCTION: Extract functions. +\& \- PROCEDURE: Extract procedures. +\& \- PACKAGE: Extract packages and package bodies. +\& \- INSERT: Extract data as INSERT statement. +\& \- COPY: Extract data as COPY statement. +\& \- PARTITION: Extract range and list Oracle partitions with subpartitions. +\& \- TYPE: Extract user defined Oracle type. +\& \- FDW: Export Oracle tables as foreign table for oracle_fdw. +\& \- MVIEW: Export materialized view. +\& \- QUERY: Try to automatically convert Oracle SQL queries. +\& \- KETTLE: Generate XML ktr template files to be used by Kettle. +\& \- DBLINK: Generate oracle foreign data wrapper server to use as dblink. +\& \- SYNONYM: Export Oracle\*(Aqs synonyms as views on other schema\*(Aqs objects. +\& \- DIRECTORY: Export Oracle\*(Aqs directories as external_file extension objects. +\& \- LOAD: Dispatch a list of queries over multiple PostgreSQl connections. +\& \- TEST: perform a diff between Oracle and PostgreSQL database. +\& \- TEST_VIEW: perform a count on both side of rows returned by views +.Ve +.Sp +Only one type of export can be perform at the same time so the \s-1TYPE\s0 directive +must be unique. If you have more than one only the last found in the file will +be registered. +.Sp +Some export type can not or should not be load directly into the PostgreSQL +database and still require little manual editing. This is the case for \s-1GRANT, +TABLESPACE, TRIGGER, FUNCTION, PROCEDURE, TYPE, QUERY\s0 and \s-1PACKAGE\s0 export types +especially if you have \s-1PLSQL\s0 code or Oracle specific \s-1SQL\s0 in it. +.Sp +For \s-1TABLESPACE\s0 you must ensure that file path exist on the system and for +\&\s-1SYNONYM\s0 you may ensure that the object's owners and schemas correspond to +the new PostgreSQL database design. +.Sp +Note that you can chained multiple export by giving to the \s-1TYPE\s0 directive a +comma-separated list of export type, but in this case you must not use \s-1COPY\s0 +or \s-1INSERT\s0 with other export type. +.Sp +Ora2Pg will convert Oracle partition using table inheritance, trigger and +functions. See document at Pg site: +http://www.postgresql.org/docs/current/interactive/ddl\-partitioning.html +.Sp +The \s-1TYPE\s0 export allow export of user defined Oracle type. If you don't use the +\&\-\-plsql command line parameter it simply dump Oracle user type asis else Ora2Pg +will try to convert it to PostgreSQL syntax. +.Sp +The \s-1KETTLE\s0 export type requires that the Oracle and PostgreSQL \s-1DNS\s0 are defined. +.Sp +Since Ora2Pg v8.1 there's three new export types: +.Sp +.Vb 7 +\& SHOW_VERSION : display Oracle version +\& SHOW_SCHEMA : display the list of schema available in the database. +\& SHOW_TABLE : display the list of tables available. +\& SHOW_COLUMN : display the list of tables columns available and the +\& Ora2PG conversion type from Oracle to PostgreSQL that will be +\& applied. It will also warn you if there\*(Aqs PostgreSQL reserved +\& words in Oracle object names. +.Ve +.Sp +Here is an example of the \s-1SHOW_COLUMN\s0 output: +.Sp +.Vb 11 +\& [2] TABLE CURRENT_SCHEMA (1 rows) (Warning: \*(AqCURRENT_SCHEMA\*(Aq is a reserved word in PostgreSQL) +\& CONSTRAINT : NUMBER(22) => bigint (Warning: \*(AqCONSTRAINT\*(Aq is a reserved word in PostgreSQL) +\& FREEZE : VARCHAR2(25) => varchar(25) (Warning: \*(AqFREEZE\*(Aq is a reserved word in PostgreSQL) +\& ... +\& [6] TABLE LOCATIONS (23 rows) +\& LOCATION_ID : NUMBER(4) => smallint +\& STREET_ADDRESS : VARCHAR2(40) => varchar(40) +\& POSTAL_CODE : VARCHAR2(12) => varchar(12) +\& CITY : VARCHAR2(30) => varchar(30) +\& STATE_PROVINCE : VARCHAR2(25) => varchar(25) +\& COUNTRY_ID : CHAR(2) => char(2) +.Ve +.Sp +Those extraction keywords are use to only display the requested information and +exit. This allows you to quickly know on what you are going to work. +.Sp +The \s-1SHOW_COLUMN\s0 allow an other ora2pg command line option: '\-\-allow relname' +or '\-a relname' to limit the displayed information to the given table. +.Sp +The \s-1SHOW_ENCODING\s0 export type will display the \s-1NLS_LANG\s0 and \s-1CLIENT_ENCODING\s0 +values that Ora2Pg will used and the real encoding of the Oracle database with +the corresponding client encoding that could be used with PostgreSQL +.Sp +Since release v8.12, Ora2Pg allow you to export your Oracle Table definition to +be use with the oracle_fdw foreign data wrapper. By using type \s-1FDW\s0 your Oracle +tables will be exported as follow: +.Sp +.Vb 5 +\& CREATE FOREIGN TABLE oratab ( +\& id integer NOT NULL, +\& text character varying(30), +\& floating double precision NOT NULL +\& ) SERVER oradb OPTIONS (table \*(AqORATAB\*(Aq); +.Ve +.Sp +Now you can use the table like a regular PostgreSQL table. +.Sp +See http://pgxn.org/dist/oracle_fdw/ for more information on this foreign data +wrapper. +.Sp +Release 10 adds a new export type destined to evaluate the content of the +database to migrate, in terms of objects and cost to end the migration: +.Sp +.Vb 1 +\& SHOW_REPORT : show a detailed report of the Oracle database content. +.Ve +.Sp +Here is a sample of report: http://ora2pg.darold.net/report.html +.Sp +There also a more advanced report with migration cost. See the dedicated chapter +about Migration Cost Evaluation. +.IP "\s-1ESTIMATE_COST\s0" 4 +.IX Item "ESTIMATE_COST" +Activate the migration cost evaluation. Must only be used with \s-1SHOW_REPORT, +FUNCTION, PROCEDURE, PACKAGE\s0 and \s-1QUERY\s0 export type. Default is disabled. +You may want to use the \-\-estimate_cost command line option instead to activate +this functionality. Note that enabling this directive will force \s-1PLSQL_PGSQL\s0 +activation. +.IP "\s-1COST_UNIT_VALUE\s0" 4 +.IX Item "COST_UNIT_VALUE" +Set the value in minutes of the migration cost evaluation unit. Default +is five minutes per unit. See \-\-cost_unit_value to change the unit value +at command line. +.IP "\s-1DUMP_AS_HTML\s0" 4 +.IX Item "DUMP_AS_HTML" +By default when using \s-1SHOW_REPORT\s0 the migration report is generated as simple +text, enabling this directive will force ora2pg to create a report in \s-1HTML\s0 +format. +.Sp +See http://ora2pg.darold.net/report.html for a sample report. +.IP "\s-1HUMAN_DAYS_LIMIT\s0" 4 +.IX Item "HUMAN_DAYS_LIMIT" +Use this directive to redefined the number of human-days limit where the +migration assessment level must switch from B to C. Default is set to 10 +human-days. +.IP "\s-1JOBS\s0" 4 +.IX Item "JOBS" +This configuration directive adds multiprocess support to \s-1COPY, FUNCTION\s0 +and \s-1PROCEDURE\s0 export type, the value is the number of process to use. +Default is multiprocess disable. +.Sp +This directive is used to set the number of cores to used to parallelize +data import into PostgreSQL. During \s-1FUNCTION\s0 or \s-1PROCEDURE\s0 export type each +function will be translated to plpgsql using a new process, the performances +gain can be very important when you have tons of function to convert. +.Sp +There's no limitation in parallel processing than the number of cores +and the PostgreSQL I/O performance capabilities. +.Sp +Doesn't work under Windows Operating System, it is simply disabled. +.IP "\s-1ORACLE_COPIES\s0" 4 +.IX Item "ORACLE_COPIES" +This configuration directive adds multiprocess support to extract data +from Oracle. The value is the number of process to use to parallelize +the select query. Default is parallel query disable. +.Sp +The parallelism is built on splitting the query following of the number +of cores given as value to \s-1ORACLE_COPIES\s0 as follow: +.Sp +.Vb 1 +\& SELECT * FROM MYTABLE WHERE ABS(MOD(COLUMN, ORACLE_COPIES)) = CUR_PROC +.Ve +.Sp +where \s-1COLUMN\s0 is a technical key like a primary or unique key where split +will be based and the current core used by the query (\s-1CUR_PROC\s0). +.Sp +Doesn't work under Windows Operating System, it is simply disabled. +.IP "\s-1DEFINED_PK\s0" 4 +.IX Item "DEFINED_PK" +This directive is used to defined the technical key to used to split +the query between number of cores set with the \s-1ORACLE_COPIES\s0 variable. +For example: +.Sp +.Vb 1 +\& DEFINED_PK EMPLOYEES:employee_id +.Ve +.Sp +The parallel query that will be used supposing that \-J or \s-1ORACLE_COPIES\s0 +is set to 8: +.Sp +.Vb 1 +\& SELECT * FROM EMPLOYEES WHERE ABS(MOD(employee_id, 8)) = N +.Ve +.Sp +where N is the current process forked starting from 0. +.IP "\s-1PARALLEL_TABLES\s0" 4 +.IX Item "PARALLEL_TABLES" +This directive is used to defined the number of tables that will be processed +in parallel for data extraction. The limit is the number of cores on your machine. +Ora2Pg will open one database connection for each parallel table extraction. +This directive, when upper than 1, will invalidate \s-1ORACLE_COPIES\s0 but not \s-1JOBS,\s0 +so the real number of process that will be used is \s-1PARALLEL_TABLES\s0 * \s-1JOBS.\s0 +.Sp +Note that this directive when set upper that 1 will also automatically enable +the \s-1FILE_PER_TABLE\s0 directive if your are exporting to files. +.IP "\s-1DEFAULT_PARALLELISM_DEGREE\s0" 4 +.IX Item "DEFAULT_PARALLELISM_DEGREE" +You can force Ora2Pg to use /*+ \s-1PARALLEL\s0(tbname, degree) */ hint in each +query used to export data from Oracle by setting a value upper than 1 to +this directive. A value of 0 or 1 disable the use of parallel hint. +Default is disabled. +.IP "\s-1FDW_SERVER\s0" 4 +.IX Item "FDW_SERVER" +This directive is used to set the name of the foreign data server that is used +in the \*(L"\s-1CREATE SERVER\s0 name \s-1FOREIGN DATA WRAPPER\s0 oracle_fdw ...\*(R" command. This +name will then be used in the \*(L"\s-1CREATE FOREIGN TABLE ...\*(R" SQL\s0 command. Default +is arbitrary set to orcl. This only concern export type \s-1FDW.\s0 +.IP "\s-1EXTERNAL_TO_FDW\s0" 4 +.IX Item "EXTERNAL_TO_FDW" +This directive, enabled by default, allow to export Oracle's External Tables as +file_fdw foreign tables. To not export these tables at all, set the directive +to 0. +.IP "\s-1INTERNAL_DATE_MAX\s0" 4 +.IX Item "INTERNAL_DATE_MAX" +Internal timestamp retrieves from custom type are extracted in the following +format: 01\-JAN\-77 12.00.00.000000 \s-1AM.\s0 It is impossible to know the exact century +that must be used, so by default any year below 49 will be added to 2000 +and others to 1900. You can use this directive to change the default value 49. +this is only relevant if you have user defined type with a column timestamp. +.IP "\s-1AUDIT_USER\s0" 4 +.IX Item "AUDIT_USER" +Set the comma separated list of username that must be used to filter +queries from the \s-1DBA_AUDIT_TRAIL\s0 table. Default is to not scan this +table and to never look for queries. This parameter is used only with +\&\s-1SHOW_REPORT\s0 and \s-1QUERY\s0 export type with no input file for queries. +Note that queries will be normalized before output unlike when a file +is given at input using the \-i option or \s-1INPUT\s0 directive. +.IP "\s-1FUNCTION_CHECK\s0" 4 +.IX Item "FUNCTION_CHECK" +Disable this directive if you want to disable check_function_bodies. +.Sp +.Vb 1 +\& SET check_function_bodies = false; +.Ve +.Sp +It disables validation of the function body string during \s-1CREATE FUNCTION.\s0 +Default is to use de postgresql.conf setting that enable it by default. +.IP "\s-1ENABLE_BLOB_EXPORT\s0" 4 +.IX Item "ENABLE_BLOB_EXPORT" +Exporting \s-1BLOB\s0 takes time, in some circumstances you may want to export +all data except the \s-1BLOB\s0 columns. In this case disable this directive and +the \s-1BLOB\s0 columns will not be included into data export. Take care that the +target bytea column do not have a \s-1NOT NULL\s0 constraint. +.IP "\s-1DATA_EXPORT_ORDER\s0" 4 +.IX Item "DATA_EXPORT_ORDER" +By default data export order will be done by sorting on table name. If you +have huge tables at end of alphabetic order and you are using multiprocess, +it can be better to set the sort order on size so that multiple small tables +can be processed before the largest tables finish. In this case set this +directive to size. Possible values are name and size. Note that export type +\&\s-1SHOW_TABLE\s0 and \s-1SHOW_COLUMN\s0 will use this sort order too, not only \s-1COPY\s0 or +\&\s-1INSERT\s0 export type. +.SS "Limiting objects to export" +.IX Subsection "Limiting objects to export" +You may want to export only a part of an Oracle database, here are a set of +configuration directives that will allow you to control what parts of the +database should be exported. +.IP "\s-1ALLOW\s0" 4 +.IX Item "ALLOW" +This directive allows you to set a list of objects on which the export must be +limited, excluding all other objects in the same type of export. The value is +a space or comma-separated list of objects name to export. You can include +valid regex into the list. For example: +.Sp +.Vb 1 +\& ALLOW EMPLOYEES SALE_.* COUNTRIES .*_GEOM_SEQ +.Ve +.Sp +will export objects with name \s-1EMPLOYEES, COUNTRIES,\s0 all objects beginning with +\&'\s-1SALE_\s0' and all objects with a name ending by '_GEOM_SEQ'. The object depends +of the export type. Note that regex will not works with 8i database, you must +use the % placeholder instead, Ora2Pg will use the \s-1LIKE\s0 operator. +.Sp +This is the manner to declare global filters that will be used with the current +export type. You can also use extended filters that will be applied on specific +objects or only on their related export type. For example: +.Sp +.Vb 1 +\& ora2pg \-p \-c ora2pg.conf \-t TRIGGER \-a \*(AqTABLE[employees]\*(Aq +.Ve +.Sp +will limit export of trigger to those defined on table employees. If you want +to extract all triggers but not some \s-1INSTEAD OF\s0 triggers: +.Sp +.Vb 1 +\& ora2pg \-c ora2pg.conf \-t TRIGGER \-e \*(AqVIEW[trg_view_.*]\*(Aq +.Ve +.Sp +Or a more complex form: +.Sp +.Vb 2 +\& ora2pg \-p \-c ora2pg.conf \-t TABLE \-a \*(AqTABLE[EMPLOYEES]\*(Aq \e +\& \-e \*(AqINDEX[emp_.*];CKEY[emp_salary_min]\*(Aq +.Ve +.Sp +This command will export the definition of the employee table but will exclude +all index beginning with 'emp_' and the \s-1CHECK\s0 constraint called 'emp_salary_min'. +.Sp +When exporting partition you can exclude some partition tables by using +.Sp +.Vb 1 +\& ora2pg \-p \-c ora2pg.conf \-t PARTITION \-e \*(AqPARTITION[PART_199.* PART_198.*]\*(Aq +.Ve +.Sp +This will exclude partitioned tables for year 1980 to 1999 from the export but +not the main partition table. The trigger will also be adapted to exclude those +table. +.Sp +With \s-1GRANT\s0 export you can use this extended form to exclude some users from the +export or limit the export to some others: +.Sp +.Vb 1 +\& ora2pg \-p \-c ora2pg.conf \-t GRANT \-a \*(AqUSER1 USER2\*(Aq +.Ve +.Sp +or +.Sp +.Vb 1 +\& ora2pg \-p \-c ora2pg.conf \-t GRANT \-a \*(AqGRANT[USER1 USER2]\*(Aq +.Ve +.Sp +will limit export grants to users \s-1USER1\s0 and \s-1USER2.\s0 But if you don't want to +export grants on some functions for these users, for example: +.Sp +.Vb 1 +\& ora2pg \-p \-c ora2pg.conf \-t GRANT \-a \*(AqUSER1 USER2\*(Aq \-e \*(AqFUNCTION[adm_.*];PROCEDURE[adm_.*]\*(Aq +.Ve +.Sp +Advanced filters may need some learning. +.Sp +Oracle doesn't allow the use of lookahead expression so you may want to exclude +some object that match the \s-1ALLOW\s0 regexp you have defined. For example if you +want to export all table starting with E but not those starting with \s-1EXP\s0 it is +not possible to do that in a single expression. This is why you can start a +regular expression with the ! character to exclude object matching the regexp +given just after. Our previous example can be written as follow: +.Sp +.Vb 1 +\& ALLOW E.* !EXP.* +.Ve +.Sp +it will be translated into: +.Sp +.Vb 1 +\& REGEXP_LIKE(..., \*(Aq^E.*$\*(Aq) AND NOT REGEXP_LIKE(..., \*(Aq^EXP.*$\*(Aq) +.Ve +.Sp +in the object search expression. +.IP "\s-1EXCLUDE\s0" 4 +.IX Item "EXCLUDE" +This directive is the opposite of the previous, it allow you to define a space +or comma-separated list of object name to exclude from the export. You can +include valid regex into the list. For example: +.Sp +.Vb 1 +\& EXCLUDE EMPLOYEES TMP_.* COUNTRIES +.Ve +.Sp +will exclude object with name \s-1EMPLOYEES, COUNTRIES\s0 and all tables beginning with +\&'tmp_'. +.Sp +For example, you can ban from export some unwanted function with this directive: +.Sp +.Vb 1 +\& EXCLUDE write_to_.* send_mail_.* +.Ve +.Sp +this example will exclude all functions, procedures or functions in a package +with the name beginning with those regex. Note that regex will not work with +8i database, you must use the % placeholder instead, Ora2Pg will use the \s-1NOT +LIKE\s0 operator. +.Sp +See above (directive '\s-1ALLOW\s0') for the extended syntax. +.IP "\s-1VIEW_AS_TABLE\s0" 4 +.IX Item "VIEW_AS_TABLE" +Set which view to export as table. By default none. Value must be a list of +view name or regexp separated by space or comma. If the object name is a view +and the export type is \s-1TABLE,\s0 the view will be exported as a create table +statement. If export type is \s-1COPY\s0 or \s-1INSERT,\s0 the corresponding data will be +exported. +.Sp +See chapter \*(L"Exporting views as PostgreSQL table\*(R" for more details. +.IP "\s-1NO_VIEW_ORDERING\s0" 4 +.IX Item "NO_VIEW_ORDERING" +By default Ora2Pg try to order views to avoid error at import time with +nested views. With a huge number of views this can take a very long time, +you can bypass this ordering by enabling this directive. +.IP "\s-1GRANT_OBJECT\s0" 4 +.IX Item "GRANT_OBJECT" +When exporting GRANTs you can specify a comma separated list of objects +for which privilege will be exported. Default is export for all objects. +Here are the possibles values \s-1TABLE, VIEW, MATERIALIZED VIEW, SEQUENCE, +PROCEDURE, FUNCTION, PACKAGE BODY, TYPE, SYNONYM, DIRECTORY.\s0 Only one object +type is allowed at a time. For example set it to \s-1TABLE\s0 if you just want to +export privilege on tables. You can use the \-g option to overwrite it. +.Sp +When used this directive prevent the export of users unless it is set to \s-1USER.\s0 +In this case only users definitions are exported. +.IP "\s-1WHERE\s0" 4 +.IX Item "WHERE" +This directive allows you to specify a \s-1WHERE\s0 clause filter when dumping the +contents of tables. Value is constructs as follows: TABLE_NAME[\s-1WHERE_CLAUSE\s0], +or if you have only one where clause for each table just put the where clause +as the value. Both are possible too. Here are some examples: +.Sp +.Vb 2 +\& # Global where clause applying to all tables included in the export +\& WHERE 1=1 +\& +\& # Apply the where clause only on table TABLE_NAME +\& WHERE TABLE_NAME[ID1=\*(Aq001\*(Aq] +\& +\& # Applies two different clause on tables TABLE_NAME and OTHER_TABLE +\& # and a generic where clause on DATE_CREATE to all other tables +\& WHERE TABLE_NAME[ID1=\*(Aq001\*(Aq OR ID1=\*(Aq002] DATE_CREATE > \*(Aq2001\-01\-01\*(Aq OTHER_TABLE[NAME=\*(Aqtest\*(Aq] +.Ve +.Sp +Any where clause not included into a table name bracket clause will be applied +to all exported table including the tables defined in the where clause. These +\&\s-1WHERE\s0 clauses are very useful if you want to archive some data or at the +opposite only export some recent data. +.Sp +To be able to quickly test data import it is useful to limit data export to the +first thousand tuples of each table. For Oracle define the following clause: +.Sp +.Vb 1 +\& WHERE ROWNUM < 1000 +.Ve +.Sp +and for MySQL, use the following: +.Sp +.Vb 1 +\& WHERE 1=1 LIMIT 1,1000 +.Ve +.Sp +This can also be restricted to some tables data export. +.IP "\s-1TOP_MAX\s0" 4 +.IX Item "TOP_MAX" +This directive is used to limit the number of item shown in the top N lists +like the top list of tables per number of rows and the top list of largest +tables in megabytes. By default it is set to 10 items. +.IP "\s-1LOG_ON_ERROR\s0" 4 +.IX Item "LOG_ON_ERROR" +Enable this directive if you want to continue direct data import on error. +When Ora2Pg received an error in the \s-1COPY\s0 or \s-1INSERT\s0 statement from PostgreSQL +it will log the statement to a file called TABLENAME_error.log in the output +directory and continue to next bulk of data. Like this you can try to fix the +statement and manually reload the error log file. Default is disabled: abort +import on error. +.IP "\s-1REPLACE_QUERY\s0" 4 +.IX Item "REPLACE_QUERY" +Sometime you may want to extract data from an Oracle table but you need a +custom query for that. Not just a \*(L"\s-1SELECT\s0 * \s-1FROM\s0 table\*(R" like Ora2Pg do +but a more complex query. This directive allows you to overwrite the query +used by Ora2Pg to extract data. The format is TABLENAME[\s-1SQL_QUERY\s0]. +If you have multiple table to extract by replacing the Ora2Pg query, you can +define multiple \s-1REPLACE_QUERY\s0 lines. +.Sp +.Vb 1 +\& REPLACE_QUERY EMPLOYEES[SELECT e.id,e.fisrtname,lastname FROM EMPLOYEES e JOIN EMP_UPDT u ON (e.id=u.id AND u.cdate>\*(Aq2014\-08\-01 00:00:00\*(Aq)] +.Ve +.SS "Control of Full Text Search export" +.IX Subsection "Control of Full Text Search export" +Several directives can be used to control the way Ora2Pg will export the +Oracle's Text search indexes. By default \s-1CONTEXT\s0 indexes will be exported +to PostgreSQL \s-1FTS\s0 indexes but \s-1CTXCAT\s0 indexes will be exported as indexes +using the pg_trgm extension. +.IP "\s-1CONTEXT_AS_TRGM\s0" 4 +.IX Item "CONTEXT_AS_TRGM" +Force Ora2Pg to translate Oracle Text indexes into PostgreSQL indexes using +pg_trgm extension. Default is to translate \s-1CONTEXT\s0 indexes into \s-1FTS\s0 indexes +and \s-1CTXCAT\s0 indexes using pg_trgm. Most of the time using pg_trgm is enough, +this is why this directive stand for. You need to create the pg_trgm extension +into the destination database before importing the objects: +.Sp +.Vb 1 +\& CREATE EXTENSION pg_trgm; +.Ve +.IP "\s-1FTS_INDEX_ONLY\s0" 4 +.IX Item "FTS_INDEX_ONLY" +By default Ora2Pg creates a function-based index to translate Oracle Text +indexes. +.Sp +.Vb 2 +\& CREATE INDEX ON t_document +\& USING gin(to_tsvector(\*(Aqpg_catalog.french\*(Aq, title)); +.Ve +.Sp +You will have to rewrite the \s-1\fBCONTAIN\s0()\fR clause using \fBto_tsvector()\fR, example: +.Sp +.Vb 2 +\& SELECT id,title FROM t_document +\& WHERE to_tsvector(title)) @@ to_tsquery(\*(Aqsearch_word\*(Aq); +.Ve +.Sp +To force Ora2Pg to create an extra tsvector column with a dedicated triggers +for \s-1FTS\s0 indexes, disable this directive. In this case, Ora2Pg will add the +column as follow: \s-1ALTER TABLE\s0 t_document \s-1ADD COLUMN\s0 tsv_title tsvector; +Then update the column to compute \s-1FTS\s0 vectors if data have been loaded before + \s-1UPDATE\s0 t_document \s-1SET\s0 tsv_title = + to_tsvector('pg_catalog.french', coalesce(title,'')); +To automatically update the column when a modification in the title column +appears, Ora2Pg adds the following trigger: +.Sp +.Vb 12 +\& CREATE FUNCTION tsv_t_document_title() RETURNS trigger AS $$ +\& BEGIN +\& IF TG_OP = \*(AqINSERT\*(Aq OR new.title != old.title THEN +\& new.tsv_title := +\& to_tsvector(\*(Aqpg_catalog.french\*(Aq, coalesce(new.title,\*(Aq\*(Aq)); +\& END IF; +\& return new; +\& END +\& $$ LANGUAGE plpgsql; +\& CREATE TRIGGER trig_tsv_t_document_title BEFORE INSERT OR UPDATE +\& ON t_document +\& FOR EACH ROW EXECUTE PROCEDURE tsv_t_document_title(); +.Ve +.Sp +When the Oracle text index is defined over multiple column, Ora2Pg will use +\&\fBsetweight()\fR to set a weight in the order of the column declaration. +.IP "\s-1FTS_CONFIG\s0" 4 +.IX Item "FTS_CONFIG" +Use this directive to force text search configuration to use. When it is not +set, Ora2Pg will autodetect the stemmer used by Oracle for each index and +pg_catalog.english if the information is not found. +.IP "\s-1USE_UNACCENT\s0" 4 +.IX Item "USE_UNACCENT" +If you want to perform your text search in an accent insensitive way, enable +this directive. Ora2Pg will create an helper function over \fBunaccent()\fR and +creates the pg_trgm indexes using this function. With \s-1FTS\s0 Ora2Pg will +redefine your text search configuration, for example: +.Sp +.Vb 3 +\& CREATE TEXT SEARCH CONFIGURATION fr (COPY = french); +\& ALTER TEXT SEARCH CONFIGURATION fr +\& ALTER MAPPING FOR hword, hword_part, word WITH unaccent, french_stem; +.Ve +.Sp +then set the \s-1FTS_CONFIG\s0 ora2pg.conf directive to fr instead of pg_catalog.english. +.Sp +When enabled, Ora2pg will create the wrapper function: +.Sp +.Vb 6 +\& CREATE OR REPLACE FUNCTION unaccent_immutable(text) +\& RETURNS text AS +\& $$ +\& SELECT public.unaccent(\*(Aqpublic.unaccent\*(Aq, $1); +\& $$ LANGUAGE sql IMMUTABLE +\& COST 1; +.Ve +.Sp +the indexes are exported as follow: +.Sp +.Vb 2 +\& CREATE INDEX t_document_title_unaccent_trgm_idx ON t_document +\& USING gin (unaccent_immutable(title) gin_trgm_ops); +.Ve +.Sp +In your queries you will need to use the same function in the search to +be able to use the function-based index. Example: +.Sp +.Vb 2 +\& SELECT * FROM t_document +\& WHERE unaccent_immutable(title) LIKE \*(Aq%donnees%\*(Aq; +.Ve +.IP "\s-1USE_LOWER_UNACCENT\s0" 4 +.IX Item "USE_LOWER_UNACCENT" +Same as above but call \fBlower()\fR in the \fBunaccent_immutable()\fR function: +.Sp +.Vb 5 +\& CREATE OR REPLACE FUNCTION unaccent_immutable(text) +\& RETURNS text AS +\& $$ +\& SELECT lower(public.unaccent(\*(Aqpublic.unaccent\*(Aq, $1)); +\& $$ LANGUAGE sql IMMUTABLE; +.Ve +.SS "Modifying object structure" +.IX Subsection "Modifying object structure" +One of the great usage of Ora2Pg is its flexibility to replicate Oracle database +into PostgreSQL database with a different structure or schema. There's three +configuration directives that allow you to map those differences. +.IP "\s-1REORDERING_COLUMNS\s0" 4 +.IX Item "REORDERING_COLUMNS" +Enable this directive to reordering columns and minimized the footprint +on disc, so that more rows fit on a data page, which is the most important +factor for speed. Default is disabled, that mean the same order than in +Oracle tables definition, that's should be enough for most usage. This +directive is only used with \s-1TABLE\s0 export. +.IP "\s-1MODIFY_STRUCT\s0" 4 +.IX Item "MODIFY_STRUCT" +This directive allows you to limit the columns to extract for a given table. The +value consist in a space-separated list of table name with a set of column +between parenthesis as follow: +.Sp +.Vb 1 +\& MODIFY_STRUCT NOM_TABLE(nomcol1,nomcol2,...) ... +.Ve +.Sp +for example: +.Sp +.Vb 1 +\& MODIFY_STRUCT T_TEST1(id,dossier) T_TEST2(id,fichier) +.Ve +.Sp +This will only extract columns 'id' and 'dossier' from table T_TEST1 and columns +\&'id' and 'fichier' from the T_TEST2 table. This directive can only be used with +\&\s-1TABLE, COPY\s0 or \s-1INSERT\s0 export. With \s-1TABLE\s0 export create table \s-1DDL\s0 will respect +the new list of columns and all indexes or foreign key pointing to or from a +column removed will not be exported. +.IP "\s-1REPLACE_TABLES\s0" 4 +.IX Item "REPLACE_TABLES" +This directive allows you to remap a list of Oracle table name to a PostgreSQL table name during export. The value is a list of space-separated values with the following structure: +.Sp +.Vb 1 +\& REPLACE_TABLES ORIG_TBNAME1:DEST_TBNAME1 ORIG_TBNAME2:DEST_TBNAME2 +.Ve +.Sp +Oracle tables \s-1ORIG_TBNAME1\s0 and \s-1ORIG_TBNAME2\s0 will be respectively renamed into +\&\s-1DEST_TBNAME1\s0 and \s-1DEST_TBNAME2\s0 +.IP "\s-1REPLACE_COLS\s0" 4 +.IX Item "REPLACE_COLS" +Like table name, the name of the column can be remapped to a different name +using the following syntax: +.Sp +.Vb 1 +\& REPLACE_COLS ORIG_TBNAME(ORIG_COLNAME1:NEW_COLNAME1,ORIG_COLNAME2:NEW_COLNAME2) +.Ve +.Sp +For example: +.Sp +.Vb 1 +\& REPLACE_COLS T_TEST(dico:dictionary,dossier:folder) +.Ve +.Sp +will rename Oracle columns 'dico' and 'dossier' from table T_TEST into new name +\&'dictionary' and 'folder'. +.IP "\s-1REPLACE_AS_BOOLEAN\s0" 4 +.IX Item "REPLACE_AS_BOOLEAN" +If you want to change the type of some Oracle columns into PostgreSQL boolean +during the export you can define here a list of tables and column separated by +space as follow. +.Sp +.Vb 1 +\& REPLACE_AS_BOOLEAN TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 TB_NAME2:COL_NAME2 +.Ve +.Sp +The values set in the boolean columns list will be replaced with the 't' and 'f' +following the default replacement values and those additionally set in directive +\&\s-1BOOLEAN_VALUES.\s0 +.Sp +Note that if you have modified the table name with \s-1REPLACE_TABLES\s0 and/or the +column's name, you need to use the name of the original table and/or column. +.Sp +.Vb 2 +\& REPLACE_COLS TB_NAME1(OLD_COL_NAME1:NEW_COL_NAME1) +\& REPLACE_AS_BOOLEAN TB_NAME1:OLD_COL_NAME1 +.Ve +.Sp +You can also give a type and a precision to automatically convert all fields of +that type as a boolean. For example: +.Sp +.Vb 1 +\& REPLACE_AS_BOOLEAN NUMBER:1 CHAR:1 TB_NAME1:COL_NAME1 TB_NAME1:COL_NAME2 +.Ve +.Sp +will also replace any field of type \fBnumber\fR\|(1) or \fBchar\fR\|(1) as a boolean in all exported +tables. +.IP "\s-1BOOLEAN_VALUES\s0" 4 +.IX Item "BOOLEAN_VALUES" +Use this to add additional definition of the possible boolean values used in +Oracle fields. You must set a space-separated list of \s-1TRUE:FALSE\s0 values. By +default here are the values recognized by Ora2Pg: +.Sp +.Vb 1 +\& BOOLEAN_VALUES yes:no y:n 1:0 true:false enabled:disabled +.Ve +.Sp +Any values defined here will be added to the default list. +.IP "\s-1REPLACE_ZERO_DATE\s0" 4 +.IX Item "REPLACE_ZERO_DATE" +When Ora2Pg find a \*(L"zero\*(R" date: 0000\-00\-00 00:00:00 it is replaced by a \s-1NULL.\s0 +This could be a problem if your column is defined with \s-1NOT NULL\s0 constraint. +If you can not remove the constraint, use this directive to set an arbitral +date that will be used instead. You can also use \-INFINITY if you don't want +to use a fake date. +.IP "\s-1INDEXES_SUFFIX\s0" 4 +.IX Item "INDEXES_SUFFIX" +Add the given value as suffix to indexes names. Useful if you have indexes +with same name as tables. For example: +.Sp +.Vb 1 +\& INDEXES_SUFFIX _idx +.Ve +.Sp +will add _idx at ed of all index name. Not so common but can help. +.IP "\s-1INDEXES_RENAMING\s0" 4 +.IX Item "INDEXES_RENAMING" +Enable this directive to rename all indexes using tablename_columns_names. +Could be very useful for database that have multiple time the same index name +or that use the same name than a table, which is not allowed by PostgreSQL +Disabled by default. +.IP "\s-1USE_INDEX_OPCLASS\s0" 4 +.IX Item "USE_INDEX_OPCLASS" +Operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops +support B\-tree indexes on the corresponding types. The difference from the +default operator classes is that the values are compared strictly character by +character rather than according to the locale-specific collation rules. This +makes these operator classes suitable for use by queries involving pattern +matching expressions (\s-1LIKE\s0 or \s-1POSIX\s0 regular expressions) when the database +does not use the standard \*(L"C\*(R" locale. If you enable, with value 1, this will +force Ora2Pg to export all indexes defined on \fBvarchar2()\fR and \fBchar()\fR columns +using those operators. If you set it to a value greater than 1 it will only +change indexes on columns where the character limit is greater or equal than +this value. For example, set it to 128 to create these kind of indexes on +columns of type varchar2(N) where N >= 128. +.IP "\s-1PREFIX_PARTITION\s0" 4 +.IX Item "PREFIX_PARTITION" +Enable this directive if you want that your partition table name will be +exported using the parent table name. Disabled by default. If you have +multiple partitioned table, when exported to PostgreSQL some partitions +could have the same name but different parent tables. This is not allowed, +table name must be unique. +.IP "\s-1PREFIX_SUB_PARTITION\s0" 4 +.IX Item "PREFIX_SUB_PARTITION" +Enable this directive if you want that your subpartition table name will be +exported using the parent partition name. Enabled by default. If the partition +names are a part of the subpartition names, you should enable this directive. +.IP "\s-1DISABLE_PARTITION\s0" 4 +.IX Item "DISABLE_PARTITION" +If you don't want to reproduce the partitioning like in Oracle and want to +export all partitioned Oracle data into the main single table in PostgreSQL +enable this directive. Ora2Pg will export all data into the main table name. +Default is to use partitioning, Ora2Pg will export data from each partition +and import them into the PostgreSQL dedicated partition table. +.IP "\s-1DISABLE_UNLOGGED\s0" 4 +.IX Item "DISABLE_UNLOGGED" +By default Ora2Pg export Oracle tables with the \s-1NOLOGGING\s0 attribute as +\&\s-1UNLOGGED\s0 tables. You may want to fully disable this feature because +you will lose all data from unlogged tables in case of a PostgreSQL crash. +Set it to 1 to export all tables as normal tables. +.SS "Oracle Spatial to PostGis" +.IX Subsection "Oracle Spatial to PostGis" +Ora2Pg fully export Spatial object from Oracle database. There's some +configuration directives that could be used to control the export. +.IP "\s-1AUTODETECT_SPATIAL_TYPE\s0" 4 +.IX Item "AUTODETECT_SPATIAL_TYPE" +By default Ora2Pg is looking at indexes to see the spatial constraint type +and dimensions defined under Oracle. Those constraints are passed as at index +creation using for example: +.Sp +.Vb 2 +\& CREATE INDEX ... INDEXTYPE IS MDSYS.SPATIAL_INDEX +\& PARAMETERS(\*(Aqsdo_indx_dims=2, layer_gtype=point\*(Aq); +.Ve +.Sp +If those Oracle constraints parameters are not set, the default is to export +those columns as generic type \s-1GEOMETRY\s0 to be able to receive any spatial type. +.Sp +The \s-1AUTODETECT_SPATIAL_TYPE\s0 directive allows to force Ora2Pg to autodetect the +real spatial type and dimension used in a spatial column otherwise a non\- +constrained \*(L"geometry\*(R" type is used. Enabling this feature will force Ora2Pg to +scan a sample of 50000 column to look at the \s-1GTYPE\s0 used. You can increase or +reduce the sample size by setting the value of \s-1AUTODETECT_SPATIAL_TYPE\s0 to the +desired number of line to scan. The directive is enabled by default. +.Sp +For example, in the case of a column named shape and defined with Oracle type +\&\s-1SDO_GEOMETRY,\s0 with \s-1AUTODETECT_SPATIAL_TYPE\s0 disabled it will be converted as: +.Sp +.Vb 1 +\& shape geometry(GEOMETRY) or shape geometry(GEOMETRYZ, 4326) +.Ve +.Sp +and if the directive is enabled and the column just contains a single +geometry type that use a single dimension: +.Sp +.Vb 1 +\& shape geometry(POLYGON, 4326) or shape geometry(POLYGONZ, 4326) +.Ve +.Sp +with a two or three dimensional polygon. +.IP "\s-1CONVERT_SRID\s0" 4 +.IX Item "CONVERT_SRID" +This directive allows you to control the automatically conversion of Oracle +\&\s-1SRID\s0 to standard \s-1EPSG.\s0 If enabled, Ora2Pg will use the Oracle function +sdo_cs.\fBmap_oracle_srid_to_epsg()\fR to convert all \s-1SRID.\s0 Enabled by default. +.Sp +If the \s-1SDO_SRID\s0 returned by Oracle is \s-1NULL,\s0 it will be replaced by the +default value 8307 converted to its \s-1EPSG\s0 value: 4326 (see \s-1DEFAULT_SRID\s0). +.Sp +If the value is upper than 1, all \s-1SRID\s0 will be forced to this value, in +this case \s-1DEFAULT_SRID\s0 will not be used when Oracle returns a null value +and the value will be forced to \s-1CONVERT_SRID.\s0 +.Sp +Note that it is also possible to set the \s-1EPSG\s0 value on Oracle side when +sdo_cs.\fBmap_oracle_srid_to_epsg()\fR return \s-1NULL\s0 if your want to force the value: +.Sp +.Vb 1 +\& system@db> UPDATE sdo_coord_ref_sys SET legacy_code=41014 WHERE srid = 27572; +.Ve +.IP "\s-1DEFAULT_SRID\s0" 4 +.IX Item "DEFAULT_SRID" +Use this directive to override the default \s-1EPSG SRID\s0 to used: 4326. +Can be overwritten by \s-1CONVERT_SRID,\s0 see above. +.IP "\s-1GEOMETRY_EXTRACT_TYPE\s0" 4 +.IX Item "GEOMETRY_EXTRACT_TYPE" +This directive can take three values: \s-1WKT\s0 (default), \s-1WKB\s0 and \s-1INTERNAL.\s0 +When it is set to \s-1WKT,\s0 Ora2Pg will use \s-1SDO_UTIL.\fBTO_WKTGEOMETRY\s0()\fR to +extract the geometry data. When it is set to \s-1WKB,\s0 Ora2Pg will use the +binary output using \s-1SDO_UTIL.\fBTO_WKBGEOMETRY\s0()\fR. If those two extract type +are calls at Oracle side, they are slow and you can easily reach Out Of +Memory when you have lot of rows. Also \s-1WKB\s0 is not able to export 3D geometry +and some geometries like \s-1CURVEPOLYGON.\s0 In this case you may use the \s-1INTERNAL\s0 +extraction type. It will use a Pure Perl library to convert the \s-1SDO_GEOMETRY\s0 +data into a \s-1WKT\s0 representation, the translation is done on Ora2Pg side. +This is a work in progress, please validate your exported data geometries +before use. Default spatial object extraction type is \s-1INTERNAL.\s0 +.IP "\s-1POSTGIS_SCHEMA\s0" 4 +.IX Item "POSTGIS_SCHEMA" +Use this directive to add a specific schema to the search path to look +for PostGis functions. +.SS "PostgreSQL Import" +.IX Subsection "PostgreSQL Import" +By default conversion to PostgreSQL format is written to file 'output.sql'. +The command: +.PP +.Vb 1 +\& psql mydb < output.sql +.Ve +.PP +will import content of file output.sql into PostgreSQL mydb database. +.IP "\s-1DATA_LIMIT\s0" 4 +.IX Item "DATA_LIMIT" +When you are performing \s-1INSERT/COPY\s0 export Ora2Pg proceed by chunks of \s-1DATA_LIMIT\s0 +tuples for speed improvement. Tuples are stored in memory before being written +to disk, so if you want speed and have enough system resources you can grow +this limit to an upper value for example: 100000 or 1000000. Before release 7.0 +a value of 0 mean no limit so that all tuples are stored in memory before being +flushed to disk. In 7.x branch this has been remove and chunk will be set to the +default: 10000 +.IP "\s-1BLOB_LIMIT\s0" 4 +.IX Item "BLOB_LIMIT" +When Ora2Pg detect a table with some \s-1BLOB\s0 it will automatically reduce the +value of this directive by dividing it by 10 until his value is below 1000. +You can control this value by setting \s-1BLOB_LIMIT.\s0 Exporting \s-1BLOB\s0 use lot of +resources, setting it to a too high value can produce \s-1OOM.\s0 +.IP "\s-1OUTPUT\s0" 4 +.IX Item "OUTPUT" +The Ora2Pg output filename can be changed with this directive. Default value is +output.sql. if you set the file name with extension .gz or .bz2 the output will +be automatically compressed. This require that the Compress::Zlib Perl module +is installed if the filename extension is .gz and that the bzip2 system command +is installed for the .bz2 extension. +.IP "\s-1OUTPUT_DIR\s0" 4 +.IX Item "OUTPUT_DIR" +Since release 7.0, you can define a base directory where the file will be written. +The directory must exists. +.IP "\s-1BZIP2\s0" 4 +.IX Item "BZIP2" +This directive allows you to specify the full path to the bzip2 program if it +can not be found in the \s-1PATH\s0 environment variable. +.IP "\s-1FILE_PER_CONSTRAINT\s0" 4 +.IX Item "FILE_PER_CONSTRAINT" +Allow object constraints to be saved in a separate file during schema export. +The file will be named \s-1CONSTRAINTS_OUTPUT,\s0 where \s-1OUTPUT\s0 is the value of the +corresponding configuration directive. You can use .gz xor .bz2 extension to +enable compression. Default is to save all data in the \s-1OUTPUT\s0 file. This +directive is usable only with \s-1TABLE\s0 export type. +.Sp +The constraints can be imported quickly into PostgreSQL using the \s-1LOAD\s0 export +type to parallelize their creation over multiple (\-j or \s-1JOBS\s0) connections. +.IP "\s-1FILE_PER_INDEX\s0" 4 +.IX Item "FILE_PER_INDEX" +Allow indexes to be saved in a separate file during schema export. The file +will be named \s-1INDEXES_OUTPUT,\s0 where \s-1OUTPUT\s0 is the value of the corresponding +configuration directive. You can use .gz xor .bz2 file extension to enable +compression. Default is to save all data in the \s-1OUTPUT\s0 file. This directive +is usable only with \s-1TABLE AND TABLESPACE\s0 export type. With the \s-1TABLESPACE\s0 +export, it is used to write \*(L"\s-1ALTER INDEX ... TABLESPACE ...\*(R"\s0 into a separate +file named \s-1TBSP_INDEXES_OUTPUT\s0 that can be loaded at end of the migration after +the indexes creation to move the indexes. +.Sp +The indexes can be imported quickly into PostgreSQL using the \s-1LOAD\s0 export +type to parallelize their creation over multiple (\-j or \s-1JOBS\s0) connections. +.IP "\s-1FILE_PER_FKEYS\s0" 4 +.IX Item "FILE_PER_FKEYS" +Allow foreign key declaration to be saved in a separate file during +schema export. By default foreign keys are exported into the main +output file or in the CONSTRAINT_output.sql file. When enabled foreign +keys will be exported into a file named FKEYS_output.sql +.IP "\s-1FILE_PER_TABLE\s0" 4 +.IX Item "FILE_PER_TABLE" +Allow data export to be saved in one file per table/view. The files will be +named as tablename_OUTPUT, where \s-1OUTPUT\s0 is the value of the corresponding +configuration directive. You can still use .gz xor .bz2 extension in the \s-1OUTPUT\s0 +directive to enable compression. Default 0 will save all data in one file, set +it to 1 to enable this feature. This is usable only during \s-1INSERT\s0 or \s-1COPY\s0 export +type. +.IP "\s-1FILE_PER_FUNCTION\s0" 4 +.IX Item "FILE_PER_FUNCTION" +Allow functions, procedures and triggers to be saved in one file per object. +The files will be named as objectname_OUTPUT. Where \s-1OUTPUT\s0 is the value of the +corresponding configuration directive. You can still use .gz xor .bz2 extension +in the \s-1OUTPUT\s0 directive to enable compression. Default 0 will save all in one +single file, set it to 1 to enable this feature. This is usable only during the +corresponding export type, the package body export has a special behavior. +.Sp +When export type is \s-1PACKAGE\s0 and you've enabled this directive, Ora2Pg will +create a directory per package, named with the lower case name of the package, +and will create one file per function/procedure into that directory. If the +configuration directive is not enabled, it will create one file per package as +packagename_OUTPUT, where \s-1OUTPUT\s0 is the value of the corresponding directive. +.IP "\s-1TRUNCATE_TABLE\s0" 4 +.IX Item "TRUNCATE_TABLE" +If this directive is set to 1, a \s-1TRUNCATE TABLE\s0 instruction will be add before +loading data. This is usable only during \s-1INSERT\s0 or \s-1COPY\s0 export type. +.Sp +When activated, the instruction will be added only if there's no global \s-1DELETE\s0 +clause or not one specific to the current table (see below). +.IP "\s-1DELETE\s0" 4 +.IX Item "DELETE" +Support for include a \s-1DELETE FROM ... WHERE\s0 clause filter before importing +data and perform a delete of some lines instead of truncating tables. +Value is construct as follow: TABLE_NAME[\s-1DELETE_WHERE_CLAUSE\s0], or +if you have only one where clause for all tables just put the delete +clause as single value. Both are possible too. Here are some examples: +.Sp +.Vb 3 +\& DELETE 1=1 # Apply to all tables and delete all tuples +\& DELETE TABLE_TEST[ID1=\*(Aq001\*(Aq] # Apply only on table TABLE_TEST +\& DELETE TABLE_TEST[ID1=\*(Aq001\*(Aq OR ID1=\*(Aq002] DATE_CREATE > \*(Aq2001\-01\-01\*(Aq TABLE_INFO[NAME=\*(Aqtest\*(Aq] +.Ve +.Sp +The last applies two different delete where clause on tables \s-1TABLE_TEST\s0 and +\&\s-1TABLE_INFO\s0 and a generic delete where clause on \s-1DATE_CREATE\s0 to all other tables. +If \s-1TRUNCATE_TABLE\s0 is enabled it will be applied to all tables not covered by +the \s-1DELETE\s0 definition. +.Sp +These \s-1DELETE\s0 clauses might be useful with regular \*(L"updates\*(R". +.IP "\s-1STOP_ON_ERROR\s0" 4 +.IX Item "STOP_ON_ERROR" +Set this parameter to 0 to not include the call to \eset \s-1ON_ERROR_STOP ON\s0 in +all \s-1SQL\s0 scripts generated by Ora2Pg. By default this order is always present +so that the script will immediately abort when an error is encountered. +.IP "\s-1COPY_FREEZE\s0" 4 +.IX Item "COPY_FREEZE" +Enable this directive to use \s-1COPY FREEZE\s0 instead of a simple \s-1COPY\s0 to +export data with rows already frozen. This is intended as a performance +option for initial data loading. Rows will be frozen only if the table +being loaded has been created or truncated in the current sub-transaction. +This will only work with export to file and when \-J or \s-1ORACLE_COPIES\s0 is +not set or default to 1. It can be used with direct import into PostgreSQL +under the same condition but \-j or \s-1JOBS\s0 must also be unset or default to 1. +.IP "\s-1CREATE_OR_REPLACE\s0" 4 +.IX Item "CREATE_OR_REPLACE" +By default Ora2Pg uses \s-1CREATE OR REPLACE\s0 in function \s-1DDL,\s0 if you need not +to override existing functions disable this configuration directive, +\&\s-1DDL\s0 will not include \s-1OR REPLACE.\s0 +.IP "\s-1NO_HEADER\s0" 4 +.IX Item "NO_HEADER" +Enabling this directive will prevent Ora2Pg to print his header into +output files. Only the translated code will be written. +.IP "\s-1PSQL_RELATIVE_PATH\s0" 4 +.IX Item "PSQL_RELATIVE_PATH" +By default Ora2Pg use \ei psql command to execute generated \s-1SQL\s0 files +if you want to use a relative path following the script execution file +enabling this option will use \eir. See psql help for more information. +.PP +When using Ora2Pg export type \s-1INSERT\s0 or \s-1COPY\s0 to dump data to file and that +\&\s-1FILE_PER_TABLE\s0 is enabled, you will be warned that Ora2Pg will not export +data again if the file already exists. This is to prevent downloading twice +table with huge amount of data. To force the download of data from these tables +you have to remove the existing output file first. +.PP +If you want to import data on the fly to the PostgreSQL database you have three +configuration directives to set the PostgreSQL database connection. This is only +possible with \s-1COPY\s0 or \s-1INSERT\s0 export type as for database schema there's no real +interest to do that. +.IP "\s-1PG_DSN\s0" 4 +.IX Item "PG_DSN" +Use this directive to set the PostgreSQL data source namespace using DBD::Pg +Perl module as follow: +.Sp +.Vb 1 +\& dbi:Pg:dbname=pgdb;host=localhost;port=5432 +.Ve +.Sp +will connect to database 'pgdb' on localhost at tcp port 5432. +.Sp +Note that this directive is only used for data export, other export need to +be imported manually through the use og psql or any other PostgreSQL client. +.IP "\s-1PG_USER\s0 and \s-1PG_PWD\s0" 4 +.IX Item "PG_USER and PG_PWD" +These two directives are used to set the login user and password. +.Sp +If you do not supply a credential with \s-1PG_PWD\s0 and you have installed the +Term::ReadKey Perl module, Ora2Pg will ask for the password interactively. If +\&\s-1PG_USER\s0 is not set it will be asked interactively too. +.IP "\s-1SYNCHRONOUS_COMMIT\s0" 4 +.IX Item "SYNCHRONOUS_COMMIT" +Specifies whether transaction commit will wait for \s-1WAL\s0 records to be written +to disk before the command returns a \*(L"success\*(R" indication to the client. This +is the equivalent to set synchronous_commit directive of postgresql.conf file. +This is only used when you load data directly to PostgreSQL, the default is +off to disable synchronous commit to gain speed at writing data. Some modified +version of PostgreSQL, like greenplum, do not have this setting, so in this +set this directive to 1, ora2pg will not try to change the setting. +.IP "\s-1PG_INITIAL_COMMAND\s0" 4 +.IX Item "PG_INITIAL_COMMAND" +This directive can be used to send an initial command to PostgreSQL, just after +the connection. For example to set some session parameters. This directive can +be used multiple times. +.SS "Column type control" +.IX Subsection "Column type control" +.IP "\s-1PG_NUMERIC_TYPE\s0" 4 +.IX Item "PG_NUMERIC_TYPE" +If set to 1 replace portable numeric type into PostgreSQL internal type. +Oracle data type \s-1NUMBER\s0(p,s) is approximatively converted to real and +float PostgreSQL data type. If you have monetary fields or don't want +rounding issues with the extra decimals you should preserve the same +numeric(p,s) PostgreSQL data type. Do that only if you need exactness +because using numeric(p,s) is slower than using real or double. +.IP "\s-1PG_INTEGER_TYPE\s0" 4 +.IX Item "PG_INTEGER_TYPE" +If set to 1 replace portable numeric type into PostgreSQL internal type. +Oracle data type \s-1NUMBER\s0(p) or \s-1NUMBER\s0 are converted to smallint, integer +or bigint PostgreSQL data type following the value of the precision. If +\&\s-1NUMBER\s0 without precision are set to \s-1DEFAULT_NUMERIC\s0 (see below). +.IP "\s-1DEFAULT_NUMERIC\s0" 4 +.IX Item "DEFAULT_NUMERIC" +\&\s-1NUMBER\s0 without precision are converted by default to bigint only if +\&\s-1PG_INTEGER_TYPE\s0 is true. You can overwrite this value to any \s-1PG\s0 type, +like integer or float. +.IP "\s-1DATA_TYPE\s0" 4 +.IX Item "DATA_TYPE" +If you're experiencing any problem in data type schema conversion with this +directive you can take full control of the correspondence between Oracle and +PostgreSQL types to redefine data type translation used in Ora2pg. The syntax +is a comma-separated list of \*(L"Oracle datatype:Postgresql datatype\*(R". Here are +the default list used: +.Sp +.Vb 1 +\& DATA_TYPE VARCHAR2:varchar,NVARCHAR2:varchar,DATE:timestamp,LONG:text,LONG RAW:bytea,CLOB:text,NCLOB:text,BLOB:bytea,BFILE:bytea,RAW:bytea,UROWID:oid,ROWID:oid,FLOAT:double precision,DEC:decimal,DECIMAL:decimal,DOUBLE PRECISION:double precision,INT:numeric,INTEGER:numeric,REAL:real,SMALLINT:smallint,BINARY_FLOAT:double precision,BINARY_DOUBLE:double precision,TIMESTAMP:timestamp,XMLTYPE:xml,BINARY_INTEGER:integer,PLS_INTEGER:integer,TIMESTAMP WITH TIME ZONE:timestamp with time zone,TIMESTAMP WITH LOCAL TIME ZONE:timestamp with time zone +.Ve +.Sp +Note that the directive and the list definition must be a single line. +.Sp +If you want to replace a type with a precision and scale you need to escape +the coma with a backslash. For example, if you want to replace all \s-1NUMBER\s0(*,0) +into bigint instead of numeric(38) add the following: +.Sp +.Vb 1 +\& DATA_TYPE NUMBER(*\e,0):bigint +.Ve +.Sp +You don't have to recopy all default type conversion but just the one you want +to rewrite. +.Sp +There's a special case with \s-1BFILE\s0 when they are converted to type \s-1TEXT,\s0 they +will just contains the full path to the external file. If you set the +destination type to \s-1BYTEA,\s0 the default, Ora2Pg will export the content of the +\&\s-1BFILE\s0 as bytea. The third case is when you set the destination type to \s-1EFILE,\s0 +in this case, Ora2Pg will export it as an \s-1EFILE\s0 record: (\s-1DIRECTORY, FILENAME\s0). +Use the \s-1DIRECTORY\s0 export type to export the existing directories as well as +privileges on those directories. +.Sp +There's no \s-1SQL\s0 function available to retrieve the path to the \s-1BFILE.\s0 Ora2Pg +have to create one using the \s-1DBMS_LOB\s0 package. +.Sp +.Vb 10 +\& CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) +\& RETURN VARCHAR2 +\& AS +\& l_dir VARCHAR2(4000); +\& l_fname VARCHAR2(4000); +\& l_path VARCHAR2(4000); +\& BEGIN +\& dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); +\& SELECT directory_path INTO l_path FROM all_directories +\& WHERE directory_name = l_dir; +\& l_dir := rtrim(l_path,\*(Aq/\*(Aq); +\& RETURN l_dir || \*(Aq/\*(Aq || l_fname; +\& END; +.Ve +.Sp +This function is only created if Ora2Pg found a table with a \s-1BFILE\s0 column and +that the destination type is \s-1TEXT.\s0 The function is dropped at the end of the +export. This concern both, \s-1COPY\s0 and \s-1INSERT\s0 export type. +.Sp +There's no \s-1SQL\s0 function available to retrieve \s-1BFILE\s0 as an \s-1EFILE\s0 record, then +Ora2Pg have to create one using the \s-1DBMS_LOB\s0 package. +.Sp +.Vb 9 +\& CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) +\& RETURN VARCHAR2 +\& AS +\& l_dir VARCHAR2(4000); +\& l_fname VARCHAR2(4000); +\& BEGIN +\& dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); +\& RETURN \*(Aq(\*(Aq || l_dir || \*(Aq,\*(Aq || l_fnamei || \*(Aq)\*(Aq; +\& END; +.Ve +.Sp +This function is only created if Ora2Pg found a table with a \s-1BFILE\s0 column and +that the destination type is \s-1EFILE.\s0 The function is dropped at the end of the +export. This concern both, \s-1COPY\s0 and \s-1INSERT\s0 export type. +.Sp +To set the destination type, use the \s-1DATA_TYPE\s0 configuration directive: +.Sp +.Vb 1 +\& DATA_TYPE BFILE:EFILE +.Ve +.Sp +for example. +.Sp +The \s-1EFILE\s0 type is a user defined type created by the PostgreSQL extension +external_file that can be found here: https://github.com/darold/external_file +This is a port of the \s-1BFILE\s0 Oracle type to PostgreSQL. +.Sp +There's no \s-1SQL\s0 function available to retrieve the content of a \s-1BFILE.\s0 Ora2Pg +have to create one using the \s-1DBMS_LOB\s0 package. +.Sp +.Vb 10 +\& CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN +\& BLOB +\& AS +\& filecontent BLOB := NULL; +\& src_file BFILE := NULL; +\& l_step PLS_INTEGER := 12000; +\& l_dir VARCHAR2(4000); +\& l_fname VARCHAR2(4000); +\& offset NUMBER := 1; +\& BEGIN +\& IF p_bfile IS NULL THEN +\& RETURN NULL; +\& END IF; +\& +\& DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); +\& src_file := BFILENAME( l_dir, l_fname ); +\& IF src_file IS NULL THEN +\& RETURN NULL; +\& END IF; +\& +\& DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); +\& DBMS_LOB.CREATETEMPORARY(filecontent, true); +\& DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); +\& DBMS_LOB.FILECLOSE(src_file); +\& RETURN filecontent; +\& END; +.Ve +.Sp +This function is only created if Ora2Pg found a table with a \s-1BFILE\s0 column and +that the destination type is bytea (the default). The function is dropped at +the end of the export. This concern both, \s-1COPY\s0 and \s-1INSERT\s0 export type. +.Sp +About the \s-1ROWID\s0 and \s-1UROWID,\s0 they are converted into \s-1OID\s0 by \*(L"logical\*(R" default +but this will through an error at data import. There is no equivalent data type +so you might want to use the \s-1DATA_TYPE\s0 directive to change the corresponding +type in PostgreSQL. You should consider replacing this data type by a bigserial +(autoincremented sequence), text or uuid data type. +.IP "\s-1MODIFY_TYPE\s0" 4 +.IX Item "MODIFY_TYPE" +Sometimes you need to force the destination type, for example a column +exported as timestamp by Ora2Pg can be forced into type date. Value is +a comma-separated list of \s-1TABLE:COLUMN:TYPE\s0 structure. If you need to use +comma or space inside type definition you will have to backslash them. +.Sp +.Vb 1 +\& MODIFY_TYPE TABLE1:COL3:varchar,TABLE1:COL4:decimal(9\e,6) +.Ve +.Sp +Type of table1.col3 will be replaced by a varchar and table1.col4 by +a decimal with precision and scale. +.Sp +If the column's type is a user defined type Ora2Pg will autodetect the +composite type and will export its data using \s-1\fBROW\s0()\fR. Some Oracle user +defined types are just array of a native type, in this case you may want +to transform this column in simple array of a PostgreSQL native type. +To do so, just redefine the destination type as wanted and Ora2Pg will +also transform the data as an array. For example, with the following +definition in Oracle: +.Sp +.Vb 7 +\& CREATE OR REPLACE TYPE mem_type IS VARRAY(10) of VARCHAR2(15); +\& CREATE TABLE club (Name VARCHAR2(10), +\& Address VARCHAR2(20), +\& City VARCHAR2(20), +\& Phone VARCHAR2(8), +\& Members mem_type +\& ); +.Ve +.Sp +custom type \*(L"mem_type\*(R" is just a string array and can be translated into +the following in PostgreSQL: +.Sp +.Vb 7 +\& CREATE TABLE club ( +\& name varchar(10), +\& address varchar(20), +\& city varchar(20), +\& phone varchar(8), +\& members text[] +\& ) ; +.Ve +.Sp +To do so, just use the directive as follow: +.Sp +.Vb 1 +\& MODIFY_TYPE CLUB:MEMBERS:text[] +.Ve +.Sp +Ora2Pg will take care to transform all data of this column in the correct +format. Only arrays of characters and numerics types are supported. +.SS "Taking export under control" +.IX Subsection "Taking export under control" +The following other configuration directives interact directly with the export process and give you fine granularity in database export control. +.IP "\s-1SKIP\s0" 4 +.IX Item "SKIP" +For \s-1TABLE\s0 export you may not want to export all schema constraints, the \s-1SKIP\s0 +configuration directive allows you to specify a space-separated list of +constraints that should not be exported. Possible values are: +.Sp +.Vb 5 +\& \- fkeys: turn off foreign key constraints +\& \- pkeys: turn off primary keys +\& \- ukeys: turn off unique column constraints +\& \- indexes: turn off all other index types +\& \- checks: turn off check constraints +.Ve +.Sp +For example: +.Sp +.Vb 1 +\& SKIP indexes,checks +.Ve +.Sp +will removed indexes and check constraints from export. +.IP "\s-1PKEY_IN_CREATE\s0" 4 +.IX Item "PKEY_IN_CREATE" +Enable this directive if you want to add primary key definition inside the +create table statement. If disabled (the default) primary key definition +will be added with an alter table statement. Enable it if you are exporting +to GreenPlum PostgreSQL database. +.IP "\s-1KEEP_PKEY_NAMES\s0" 4 +.IX Item "KEEP_PKEY_NAMES" +By default names of the primary and unique key in the source Oracle database +are ignored and key names are autogenerated in the target PostgreSQL database +with the PostgreSQL internal default naming rules. If you want to preserve +Oracle primary and unique key names set this option to 1. +.IP "\s-1FKEY_ADD_UPDATE\s0" 4 +.IX Item "FKEY_ADD_UPDATE" +This directive allows you to add an \s-1ON UPDATE CASCADE\s0 option to a foreign +key when a \s-1ON DELETE CASCADE\s0 is defined or always. Oracle do not support +this feature, you have to use trigger to operate the \s-1ON UPDATE CASCADE.\s0 +As PostgreSQL has this feature, you can choose how to add the foreign +key option. There are three values to this directive: never, the default +that mean that foreign keys will be declared exactly like in Oracle. +The second value is delete, that mean that the \s-1ON UPDATE CASCADE\s0 option +will be added only if the \s-1ON DELETE CASCADE\s0 is already defined on the +foreign Keys. The last value, always, will force all foreign keys to be +defined using the update option. +.IP "\s-1FKEY_DEFERRABLE\s0" 4 +.IX Item "FKEY_DEFERRABLE" +When exporting tables, Ora2Pg normally exports constraints as they are, if they +are non-deferrable they are exported as non-deferrable. However, non-deferrable +constraints will probably cause problems when attempting to import data to Pg. +The \s-1FKEY_DEFERRABLE\s0 option set to 1 will cause all foreign key constraints to +be exported as deferrable. +.IP "\s-1DEFER_FKEY\s0" 4 +.IX Item "DEFER_FKEY" +In addition to exporting data when the \s-1DEFER_FKEY\s0 option set to 1, it will add +a command to defer all foreign key constraints during data export and +the import will be done in a single transaction. This will work only if +foreign keys have been exported as deferrable and you are not using direct +import to PostgreSQL (\s-1PG_DSN\s0 is not defined). Constraints will then be +checked at the end of the transaction. +.Sp +This directive can also be enabled if you want to force all foreign keys +to be created as deferrable and initially deferred during schema export +(\s-1TABLE\s0 export type). +.IP "\s-1DROP_FKEY\s0" 4 +.IX Item "DROP_FKEY" +If deferring foreign keys is not possible due to the amount of data in a +single transaction, you've not exported foreign keys as deferrable or you +are using direct import to PostgreSQL, you can use the \s-1DROP_FKEY\s0 directive. +.Sp +It will drop all foreign keys before all data import and recreate them at +the end of the import. +.IP "\s-1DROP_INDEXES\s0" 4 +.IX Item "DROP_INDEXES" +This directive allows you to gain lot of speed improvement during data import +by removing all indexes that are not an automatic index (indexes of primary +keys) and recreate them at the end of data import. Of course it is far better +to not import indexes and constraints before having imported all data. +.IP "\s-1DISABLE_TRIGGERS\s0" 4 +.IX Item "DISABLE_TRIGGERS" +This directive is used to disable triggers on all tables in \s-1COPY\s0 or \s-1INSERT\s0 +export modes. Available values are \s-1USER\s0 (disable user-defined triggers only) +and \s-1ALL\s0 (includes \s-1RI\s0 system triggers). Default is 0: do not add \s-1SQL\s0 statements +to disable trigger before data import. +.Sp +If you want to disable triggers during data migration, set the value to +\&\s-1USER\s0 if your are connected as non superuser and \s-1ALL\s0 if you are connected +as PostgreSQL superuser. A value of 1 is equal to \s-1USER.\s0 +.IP "\s-1DISABLE_SEQUENCE\s0" 4 +.IX Item "DISABLE_SEQUENCE" +If set to 1 it disables alter of sequences on all tables during \s-1COPY\s0 or \s-1INSERT\s0 export +mode. This is used to prevent the update of sequence during data migration. +Default is 0, alter sequences. +.IP "\s-1NOESCAPE\s0" 4 +.IX Item "NOESCAPE" +By default all data that are not of type date or time are escaped. If you +experience any problem with that you can set it to 1 to disable character +escaping during data export. This directive is only used during a \s-1COPY\s0 export. +See \s-1STANDARD_CONFORMING_STRINGS\s0 for enabling/disabling escape with \s-1INSERT\s0 +statements. +.IP "\s-1STANDARD_CONFORMING_STRINGS\s0" 4 +.IX Item "STANDARD_CONFORMING_STRINGS" +This controls whether ordinary string literals ('...') treat backslashes +literally, as specified in \s-1SQL\s0 standard. This was the default before Ora2Pg +v8.5 so that all strings was escaped first, now this is currently on, causing +Ora2Pg to use the escape string syntax (E'...') if this parameter is not +set to 0. This is the exact behavior of the same option in PostgreSQL. +This directive is only used during data export to build \s-1INSERT\s0 statements. +See \s-1NOESCAPE\s0 for enabling/disabling escape in \s-1COPY\s0 statements. +.IP "\s-1TRIM_TYPE\s0" 4 +.IX Item "TRIM_TYPE" +If you want to convert \s-1CHAR\s0(n) from Oracle into varchar(n) or text on PostgreSQL +using directive \s-1DATA_TYPE,\s0 you might want to do some trimming on the data. By +default Ora2Pg will auto-detect this conversion and remove any whitespace at both +leading and trailing position. If you just want to remove the leadings character +set the value to \s-1LEADING.\s0 If you just want to remove the trailing character, set +the value to \s-1TRAILING.\s0 Default value is \s-1BOTH.\s0 +.IP "\s-1TRIM_CHAR\s0" 4 +.IX Item "TRIM_CHAR" +The default trimming character is space, use this directive if you need to +change the character that will be removed. For example, set it to \- if you +have leading \- in the char(n) field. To use space as trimming charger, comment +this directive, this is the default value. +.IP "\s-1PRESERVE_CASE\s0" 4 +.IX Item "PRESERVE_CASE" +If you want to preserve the case of Oracle object name set this directive to 1. +By default Ora2Pg will convert all Oracle object names to lower case. I do not +recommend to enable this unless you will always have to double-quote object +names on all your \s-1SQL\s0 scripts. +.IP "\s-1ORA_RESERVED_WORDS\s0" 4 +.IX Item "ORA_RESERVED_WORDS" +Allow escaping of column name using Oracle reserved words. Value is a list of +comma-separated reserved word. Default: audit,comment,references. +.IP "\s-1USE_RESERVED_WORDS\s0" 4 +.IX Item "USE_RESERVED_WORDS" +Enable this directive if you have table or column names that are a reserved +word for PostgreSQL. Ora2Pg will double quote the name of the object. +.IP "\s-1GEN_USER_PWD\s0" 4 +.IX Item "GEN_USER_PWD" +Set this directive to 1 to replace default password by a random password for all +extracted user during a \s-1GRANT\s0 export. +.IP "\s-1PG_SUPPORTS_MVIEW\s0" 4 +.IX Item "PG_SUPPORTS_MVIEW" +Since PostgreSQL 9.3, materialized view are supported with the \s-1SQL\s0 syntax +\&'\s-1CREATE MATERIALIZED VIEW\s0'. To force Ora2Pg to use the native PostgreSQL +support you must enable this configuration \- enable by default. If you want +to use the old style with table and a set of function, you should disable it. +.IP "\s-1PG_SUPPORTS_IFEXISTS\s0" 4 +.IX Item "PG_SUPPORTS_IFEXISTS" +PostgreSQL version below 9.x do not support \s-1IF EXISTS\s0 in \s-1DDL\s0 statements. +Disabling the directive with value 0 will prevent Ora2Pg to add those +keywords in all generated statements. Default value is 1, enabled. +.IP "\s-1PG_SUPPORTS_ROLE\s0 (Deprecated)" 4 +.IX Item "PG_SUPPORTS_ROLE (Deprecated)" +This option is deprecated since Ora2Pg release v7.3. +.Sp +By default Oracle roles are translated into PostgreSQL groups. If you have +PostgreSQL 8.1 or more consider the use of \s-1ROLES\s0 and set this directive to 1 +to export roles. +.IP "\s-1PG_SUPPORTS_INOUT\s0 (Deprecated)" 4 +.IX Item "PG_SUPPORTS_INOUT (Deprecated)" +This option is deprecated since Ora2Pg release v7.3. +.Sp +If set to 0, all \s-1IN, OUT\s0 or \s-1INOUT\s0 parameters will not be used into the generated +PostgreSQL function declarations (disable it for PostgreSQL database version +lower than 8.1), This is now enable by default. +.IP "\s-1PG_SUPPORTS_DEFAULT\s0" 4 +.IX Item "PG_SUPPORTS_DEFAULT" +This directive enable or disable the use of default parameter value in function +export. Until PostgreSQL 8.4 such a default value was not supported, this feature +is now enable by default. +.IP "\s-1PG_SUPPORTS_WHEN\s0 (Deprecated)" 4 +.IX Item "PG_SUPPORTS_WHEN (Deprecated)" +Add support to \s-1WHEN\s0 clause on triggers as PostgreSQL v9.0 now support it. This +directive is enabled by default, set it to 0 disable this feature. +.IP "\s-1PG_SUPPORTS_INSTEADOF\s0 (Deprecated)" 4 +.IX Item "PG_SUPPORTS_INSTEADOF (Deprecated)" +Add support to \s-1INSTEAD OF\s0 usage on triggers (used with \s-1PG\s0 >= 9.1), if this +directive is disabled the \s-1INSTEAD OF\s0 triggers will be rewritten as Pg rules. +.IP "\s-1PG_SUPPORTS_CHECKOPTION\s0" 4 +.IX Item "PG_SUPPORTS_CHECKOPTION" +When enabled, export views with \s-1CHECK OPTION.\s0 Disable it if you have PostgreSQL +version prior to 9.4. Default: 1, enabled. +.IP "\s-1PG_SUPPORTS_IFEXISTS\s0" 4 +.IX Item "PG_SUPPORTS_IFEXISTS" +If disabled, do not export object with \s-1IF EXISTS\s0 statements. +Enabled by default. +.IP "\s-1PG_SUPPORTS_PARTITION\s0" 4 +.IX Item "PG_SUPPORTS_PARTITION" +PostgreSQL version prior to 10.0 do not have native partitioning. +Enable this directive if you want to use declarative partitioning. +Enable by default. +.IP "\s-1PG_SUPPORTS_SUBSTR\s0" 4 +.IX Item "PG_SUPPORTS_SUBSTR" +Some versions of PostgreSQL like Redshift doesn't support \fBsubstr()\fR +and it need to be replaced by a call to \fBsubstring()\fR. In this case, +disable it. +.IP "\s-1PG_SUPPORTS_NAMED_OPERATOR\s0" 4 +.IX Item "PG_SUPPORTS_NAMED_OPERATOR" +Disable this directive if you are using \s-1PG\s0 < 9.5, \s-1PL/SQL\s0 operator used in +named parameter => will be replaced by PostgreSQL proprietary operator := +Enable by default. +.IP "\s-1PG_SUPPORTS_IDENTITY\s0" 4 +.IX Item "PG_SUPPORTS_IDENTITY" +Enable this directive if you have PostgreSQL >= 10 to use \s-1IDENTITY\s0 columns +instead of serial or bigserial data type. If \s-1PG_SUPPORTS_IDENTITY\s0 is disabled +and there is \s-1IDENTITY\s0 column in the Oracle table, they are exported as serial +or bigserial columns. When it is enabled they are exported as \s-1IDENTITY\s0 columns +like: +.Sp +.Vb 4 +\& CREATE TABLE identity_test_tab ( +\& id bigint GENERATED ALWAYS AS IDENTITY, +\& description varchar(30) +\& ) ; +.Ve +.Sp +If there is non default sequence options set in Oracle, they will be appended +after the \s-1IDENTITY\s0 keyword. +Additionally in both cases, Ora2Pg will create a file AUTOINCREMENT_output.sql +with a embedded function to update the associated sequences with the restart +value set to \*(L"\s-1SELECT\s0 max(colname)+1 \s-1FROM\s0 tablename\*(R". Of course this file must +be imported after data import otherwise sequence will be kept to start value. +Enabled by default. +.IP "\s-1PG_SUPPORTS_PROCEDURE\s0" 4 +.IX Item "PG_SUPPORTS_PROCEDURE" +PostgreSQL v11 adds support of \s-1PROCEDURE,\s0 enable it if you use such version. +.IP "\s-1BITMAP_AS_GIN\s0" 4 +.IX Item "BITMAP_AS_GIN" +Use btree_gin extension to create bitmap like index with pg >= 9.4 +You will need to create the extension by yourself: + create extension btree_gin; +Default is to create \s-1GIN\s0 index, when disabled, a btree index will be created +.IP "\s-1PG_BACKGROUND\s0" 4 +.IX Item "PG_BACKGROUND" +Use pg_background extension to create an autonomous transaction instead +of using a dblink wrapper. With pg >= 9.5 only. Default is to use dblink. +See https://github.com/vibhorkum/pg_background about this extension. +.IP "\s-1DBLINK_CONN\s0" 4 +.IX Item "DBLINK_CONN" +By default if you have an autonomous transaction translated using dblink +extension instead of pg_background the connection is defined using the +values set with \s-1PG_DSN, PG_USER\s0 and \s-1PG_PWD.\s0 If you want to fully override +the connection string use this directive as follow to set the connection +in the autonomous transaction wrapper function. For example: +.Sp +.Vb 1 +\& DBLINK_CONN port=5432 dbname=pgdb host=localhost user=pguser password=pgpass +.Ve +.IP "\s-1LONGREADLEN\s0" 4 +.IX Item "LONGREADLEN" +Use this directive to set the database handle's 'LongReadLen' attribute to a +value that will be the larger than the expected size of the LOBs. The default +is 1MB witch may not be enough to extract BLOBs or CLOBs. If the size of the +\&\s-1LOB\s0 exceeds the 'LongReadLen' DBD::Oracle will return a '\s-1ORA\-24345: A\s0 Truncation' +error. Default: 1023*1024 bytes. +.Sp +Take a look at this page to learn more: http://search.cpan.org/~pythian/DBD\-Oracle\-1.22/Oracle.pm#Data_Interface_for_Persistent_LOBs +.Sp +Important note: If you increase the value of this directive take care that +\&\s-1DATA_LIMIT\s0 will probably needs to be reduced. Even if you only have a 1MB blob, +trying to read 10000 of them (the default \s-1DATA_LIMIT\s0) all at once will require +10GB of memory. You may extract data from those table separately and set a +\&\s-1DATA_LIMIT\s0 to 500 or lower, otherwise you may experience some out of memory. +.IP "\s-1LONGTRUNKOK\s0" 4 +.IX Item "LONGTRUNKOK" +If you want to bypass the '\s-1ORA\-24345: A\s0 Truncation' error, set this directive +to 1, it will truncate the data extracted to the LongReadLen value. Disable +by default so that you will be warned if your LongReadLen value is not high +enough. +.IP "\s-1USE_LOB_LOCATOR\s0" 4 +.IX Item "USE_LOB_LOCATOR" +Disable this if you want to load full content of \s-1BLOB\s0 and \s-1CLOB\s0 and not use +\&\s-1LOB\s0 locators. In this case you will have to set \s-1LONGREADLEN\s0 to the right +value. Note that this will not improve speed of \s-1BLOB\s0 export as most of the +time is always consumed by the bytea escaping and in this case export is +done line by line and not by chunk of \s-1DATA_LIMIT\s0 rows. For more information +on how it works, see http://search.cpan.org/~pythian/DBD\-Oracle\-1.74/lib/DBD/Oracle.pm#Data_Interface_for_LOB_Locators +.Sp +Default is enabled, it use \s-1LOB\s0 locators. +.IP "\s-1LOB_CHUNK_SIZE\s0" 4 +.IX Item "LOB_CHUNK_SIZE" +Oracle recommends reading from and writing to a \s-1LOB\s0 in batches using a +multiple of the \s-1LOB\s0 chunk size. This chunk size defaults to 8k (8192). +Recent tests shown that the best performances can be reach with higher +value like 512K or 4Mb. +.Sp +A quick benchmark with 30120 rows with different size of \s-1BLOB\s0 (200x5Mb, +19800x212k, 10000x942K, 100x17Mb, 20x156Mb), with DATA_LIMIT=100, +LONGREADLEN=170Mb and a total table size of 20GB gives: +.Sp +.Vb 4 +\& no lob locator : 22m46,218s (1365 sec., avg: 22 recs/sec) +\& chunk size 8k : 15m50,886s (951 sec., avg: 31 recs/sec) +\& chunk size 512k : 1m28,161s (88 sec., avg: 342 recs/sec) +\& chunk size 4Mb : 1m23,717s (83 sec., avg: 362 recs/sec) +.Ve +.Sp +In conclusion it can be more than 10 time faster with \s-1LOB_CHUNK_SIZE\s0 set +to 4Mb. Depending of the size of most \s-1BLOB\s0 you may want to adjust the value +here. For example if you have a majority of small lobs bellow 8K, using 8192 +is better to not waste space. Default value for \s-1LOB_CHUNK_SIZE\s0 is 512000. +.IP "\s-1XML_PRETTY\s0" 4 +.IX Item "XML_PRETTY" +Force the use \fBgetStringVal()\fR instead of \fBgetClobVal()\fR for \s-1XML\s0 data export. Default is 1, +enabled for backward compatibility. Set it to 0 to use extract method a la \s-1CLOB.\s0 +Note that \s-1XML\s0 value extracted with \fBgetStringVal()\fR must not exceed \s-1VARCHAR2\s0 size +limit (4000) otherwise it will return an error. +.IP "\s-1ENABLE_MICROSECOND\s0" 4 +.IX Item "ENABLE_MICROSECOND" +Set it to O if you want to disable export of millisecond from Oracle timestamp +columns. By default milliseconds are exported with the use of following format: +.Sp +.Vb 1 +\& \*(AqYYYY\-MM\-DD HH24:MI:SS.FF\*(Aq +.Ve +.Sp +Disabling will force the use of the following Oracle format: +.Sp +.Vb 1 +\& to_char(..., \*(AqYYYY\-MM\-DD HH24:MI:SS\*(Aq) +.Ve +.Sp +By default milliseconds are exported. +.IP "\s-1DISABLE_COMMENT\s0" 4 +.IX Item "DISABLE_COMMENT" +Set this to 1 if you don't want to export comment associated to tables and +columns definition. Default is enabled. +.SS "Control MySQL export behavior" +.IX Subsection "Control MySQL export behavior" +.IP "\s-1MYSQL_PIPES_AS_CONCAT\s0" 4 +.IX Item "MYSQL_PIPES_AS_CONCAT" +Enable this if double pipe and double ampersand (|| and &&) should not be +taken as equivalent to \s-1OR\s0 and \s-1AND.\s0 It depend of the variable \f(CW@sql_mode\fR, +Use it only if Ora2Pg fail on auto detecting this behavior. +.IP "\s-1MYSQL_INTERNAL_EXTRACT_FORMAT\s0" 4 +.IX Item "MYSQL_INTERNAL_EXTRACT_FORMAT" +Enable this directive if you want \s-1\fBEXTRACT\s0()\fR replacement to use the internal +format returned as an integer, for example \s-1DD HH24:MM:SS\s0 will be replaced +with format; DDHH24MMSS::bigint, this depend of your apps usage. +.SS "Special options to handle character encoding" +.IX Subsection "Special options to handle character encoding" +.IP "\s-1NLS_LANG\s0 and \s-1NLS_NCHAR\s0" 4 +.IX Item "NLS_LANG and NLS_NCHAR" +By default Ora2Pg will set \s-1NLS_LANG\s0 to \s-1AMERICAN_AMERICA.AL32UTF8\s0 and \s-1NLS_NCHAR\s0 +to \s-1AL32UTF8.\s0 It is not recommended to change those settings but in some case it +could be useful. Using your own settings with those configuration directive will +change the client encoding at Oracle side by setting the environment variables +\&\f(CW$ENV\fR{\s-1NLS_LANG\s0} and \f(CW$ENV\fR{\s-1NLS_NCHAR\s0}. +.IP "\s-1BINMODE\s0" 4 +.IX Item "BINMODE" +By default Ora2Pg will force Perl to use utf8 I/O encoding. This is done through +a call to the Perl pragma: +.Sp +.Vb 1 +\& use open \*(Aq:utf8\*(Aq; +.Ve +.Sp +You can override this encoding by using the \s-1BINMODE\s0 directive, for example you +can set it to :locale to use your locale or iso\-8859\-7, it will respectively use +.Sp +.Vb 2 +\& use open \*(Aq:locale\*(Aq; +\& use open \*(Aq:encoding(iso\-8859\-7)\*(Aq; +.Ve +.Sp +If you have change the \s-1NLS_LANG\s0 in non \s-1UTF8\s0 encoding, you might want to set this +directive. See http://perldoc.perl.org/5.14.2/open.html for more information. +Most of the time, leave this directive commented. +.IP "\s-1CLIENT_ENCODING\s0" 4 +.IX Item "CLIENT_ENCODING" +By default PostgreSQL client encoding is automatically set to \s-1UTF8\s0 to avoid +encoding issue. If you have changed the value of \s-1NLS_LANG\s0 you might have to +change the encoding of the PostgreSQL client. +.Sp +You can take a look at the PostgreSQL supported character sets here: http://www.postgresql.org/docs/9.0/static/multibyte.html +.SS "\s-1PLSQL\s0 to \s-1PLPGSQL\s0 conversion" +.IX Subsection "PLSQL to PLPGSQL conversion" +Automatic code conversion from Oracle \s-1PLSQL\s0 to PostgreSQL \s-1PLPGSQL\s0 is a work in +progress in Ora2Pg and surely you will always have manual work. The Perl code +used for automatic conversion is all stored in a specific Perl Module named +Ora2Pg/PLSQL.pm feel free to modify/add you own code and send me patches. The +main work in on function, procedure, package and package body headers and +parameters rewrite. +.IP "\s-1PLSQL_PGSQL\s0" 4 +.IX Item "PLSQL_PGSQL" +Enable/disable \s-1PLSQL\s0 to \s-1PLPGSQL\s0 conversion. Enabled by default. +.IP "\s-1NULL_EQUAL_EMPTY\s0" 4 +.IX Item "NULL_EQUAL_EMPTY" +Ora2Pg can replace all conditions with a test on \s-1NULL\s0 by a call to the +\&\fBcoalesce()\fR function to mimic the Oracle behavior where empty string are +considered equal to \s-1NULL.\s0 +.Sp +.Vb 2 +\& (field1 IS NULL) is replaced by (coalesce(field1::text, \*(Aq\*(Aq) = \*(Aq\*(Aq) +\& (field2 IS NOT NULL) is replaced by (field2 IS NOT NULL AND field2::text <> \*(Aq\*(Aq) +.Ve +.Sp +You might want this replacement to be sure that your application will have the +same behavior but if you have control on you application a better way is to +change it to transform empty string into \s-1NULL\s0 because PostgreSQL makes the +difference. +.IP "\s-1EMPTY_LOB_NULL\s0" 4 +.IX Item "EMPTY_LOB_NULL" +Force \fBempty_clob()\fR and \fBempty_blob()\fR to be exported as \s-1NULL\s0 instead as empty +string for the first one and '\ex' for the second. If \s-1NULL\s0 is allowed in your +column this might improve data export speed if you have lot of empty lob. +Default is to preserve the exact data from Oracle. +.IP "\s-1PACKAGE_AS_SCHEMA\s0" 4 +.IX Item "PACKAGE_AS_SCHEMA" +If you don't want to export package as schema but as simple functions you +might also want to replace all call to package_name.function_name. If you +disable the \s-1PACKAGE_AS_SCHEMA\s0 directive then Ora2Pg will replace all call +to package_name.\fBfunction_name()\fR by \fBpackage_name_function_name()\fR. Default +is to use a schema to emulate package. +.Sp +The replacement will be done in all kind of \s-1DDL\s0 or code that is parsed by +the \s-1PLSQL\s0 to \s-1PLPGSQL\s0 converter. \s-1PLSQL_PGSQL\s0 must be enabled or \-p used in +command line. +.IP "\s-1REWRITE_OUTER_JOIN\s0" 4 +.IX Item "REWRITE_OUTER_JOIN" +Enable this directive if the rewrite of Oracle native syntax (+) of +\&\s-1OUTER JOIN\s0 is broken. This will force Ora2Pg to not rewrite such code, +default is to try to rewrite simple form of right outer join for the +moment. +.IP "\s-1UUID_FUNCTION\s0" 4 +.IX Item "UUID_FUNCTION" +By default Ora2Pg will convert call to \s-1\fBSYS_GUID\s0()\fR Oracle function +with a call to uuid_generate_v4 from uuid-ossp extension. You can +redefined it to use the gen_random_uuid function from pgcrypto +extension by changing the function name. Default to uuid_generate_v4. +.Sp +Note that when a \s-1RAW\s0(n) column has \*(L"\s-1\fBSYS_GUID\s0()\fR\*(R" as default value +Ora2Pg will automatically translate the type of the column into uuid +which might be the right translation in most of the case. +.IP "\s-1FUNCTION_STABLE\s0" 4 +.IX Item "FUNCTION_STABLE" +By default Oracle functions are marked as \s-1STABLE\s0 as they can not modify data +unless when used in \s-1PL/SQL\s0 with variable assignment or as conditional +expression. You can force Ora2Pg to create these function as \s-1VOLATILE\s0 by +disabling this configuration directive. +.IP "\s-1COMMENT_COMMIT_ROLLBACK\s0" 4 +.IX Item "COMMENT_COMMIT_ROLLBACK" +By default call to \s-1COMMIT/ROLLBACK\s0 are kept untouched by Ora2Pg to force +the user to review the logic of the function. Once it is fixed in Oracle +source code or you want to comment this calls enable the following directive. +.IP "\s-1COMMENT_SAVEPOINT\s0" 4 +.IX Item "COMMENT_SAVEPOINT" +It is common to see \s-1SAVEPOINT\s0 call inside \s-1PL/SQL\s0 procedure together with +a \s-1ROLLBACK TO\s0 savepoint_name. When \s-1COMMENT_COMMIT_ROLLBACK\s0 is enabled you +may want to also comment \s-1SAVEPOINT\s0 calls, in this case enable it. +.IP "\s-1STRING_CONSTANT_REGEXP\s0" 4 +.IX Item "STRING_CONSTANT_REGEXP" +Ora2Pg replace all string constant during the pl/sql to plpgsql translation, +string constant are all text include between single quote. If you have some +string placeholder used in dynamic call to queries you can set a list of +regexp to be temporary replaced to not break the parser. For example: +.Sp +.Vb 1 +\& STRING_CONSTANT_REGEXP +.Ve +.Sp +The list of regexp must use the semi colon as separator. +.IP "\s-1ALTERNATIVE_QUOTING_REGEXP\s0" 4 +.IX Item "ALTERNATIVE_QUOTING_REGEXP" +To support the Alternative Quoting Mechanism ('Q' or 'q') for String Literals +set the regexp with the text capture to use to extract the text part. For +example with a variable declared as +.Sp +.Vb 1 +\& c_sample VARCHAR2(100 CHAR) := q\*(Aq{This doesn\*(Aqt work.}\*(Aq; +.Ve +.Sp +the regexp to use must be: +.Sp +.Vb 1 +\& ALTERNATIVE_QUOTING_REGEXP q\*(Aq{(.*)}\*(Aq +.Ve +.Sp +ora2pg will use the $$ delimiter, with the example the result will be: +.Sp +.Vb 1 +\& c_sample varchar(100) := $$This doesn\*(Aqt work.$$; +.Ve +.Sp +The value of this configuration directive can be a list of regexp +separated by a semi colon. The capture part (between parenthesis) is +mandatory in each regexp if you want to restore the string constant. +.IP "\s-1USE_ORAFCE\s0" 4 +.IX Item "USE_ORAFCE" +If you want to use functions defined in the Orafce library and prevent +Ora2Pg to translate call to these functions, enable this directive. +The Orafce library can be found here: https://github.com/orafce/orafce +.Sp +By default Ora2pg rewrite \fBadd_month()\fR, \fBadd_year()\fR, \fBdate_trunc()\fR and +\&\fBto_char()\fR functions, but you may prefer to use the orafce version of +these function that do not need any code transformation. +.IP "\s-1AUTONOMOUS_TRANSACTION\s0" 4 +.IX Item "AUTONOMOUS_TRANSACTION" +Enable translation of autonomous transactions into a wrapper function +using dblink or pg_background extension. If you don't want to use this +translation and just want the function to be exported as a normal one +without the pragma call, disable this directive. +.SS "Materialized view" +.IX Subsection "Materialized view" +Materialized views are exported as snapshot \*(L"Snapshot Materialized Views\*(R" as +PostgreSQL only supports full refresh. +.PP +If you want to import the materialized views in PostgreSQL prior to 9.3 you +have to set configuration directive \s-1PG_SUPPORTS_MVIEW\s0 to 0. In this case +Ora2Pg will export all materialized views as explain in this document: +.PP +.Vb 1 +\& http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views. +.Ve +.PP +When exporting materialized view Ora2Pg will first add the \s-1SQL\s0 code to create the \*(L"materialized_views\*(R" table: +.PP +.Vb 6 +\& CREATE TABLE materialized_views ( +\& mview_name text NOT NULL PRIMARY KEY, +\& view_name text NOT NULL, +\& iname text, +\& last_refresh TIMESTAMP WITH TIME ZONE +\& ); +.Ve +.PP +all materialized views will have an entry in this table. It then adds the +plpgsql code to create tree functions: +.PP +.Vb 3 +\& create_materialized_view(text, text, text) used to create a materialized view +\& drop_materialized_view(text) used to delete a materialized view +\& refresh_full_materialized_view(text) used to refresh a view +.Ve +.PP +then it adds the \s-1SQL\s0 code to create the view and the materialized view: +.PP +.Vb 2 +\& CREATE VIEW mviewname_mview AS +\& SELECT ... FROM ...; +\& +\& SELECT create_materialized_view(\*(Aqmviewname\*(Aq,\*(Aqmviewname_mview\*(Aq, change with the name of the column to used for the index); +.Ve +.PP +The first argument is the name of the materialized view, the second the name of +the view on which the materialized view is based and the third is the column +name on which the index should be build (aka most of the time the primary key). +This column is not automatically deduced so you need to replace its name. +.PP +As said above Ora2Pg only supports snapshot materialized views so the table will +be entirely refreshed by issuing first a truncate of the table and then by load +again all data from the view: +.PP +.Vb 1 +\& refresh_full_materialized_view(\*(Aqmviewname\*(Aq); +.Ve +.PP +To drop the materialized view you just have to call the \fBdrop_materialized_view()\fR +function with the name of the materialized view as parameter. +.SS "Other configuration directives" +.IX Subsection "Other configuration directives" +.IP "\s-1DEBUG\s0" 4 +.IX Item "DEBUG" +Set it to 1 will enable verbose output. +.IP "\s-1IMPORT\s0" 4 +.IX Item "IMPORT" +You can define common Ora2Pg configuration directives into a single file that +can be imported into other configuration files with the \s-1IMPORT\s0 configuration +directive as follow: +.Sp +.Vb 1 +\& IMPORT commonfile.conf +.Ve +.Sp +will import all configuration directives defined into commonfile.conf into the +current configuration file. +.SS "Exporting views as PostgreSQL tables" +.IX Subsection "Exporting views as PostgreSQL tables" +You can export any Oracle view as a PostgreSQL table simply by setting \s-1TYPE\s0 +configuration option to \s-1TABLE\s0 to have the corresponding create table statement. +Or use type \s-1COPY\s0 or \s-1INSERT\s0 to export the corresponding data. To allow that you +have to specify your views in the \s-1VIEW_AS_TABLE\s0 configuration option. +.PP +Then if Ora2Pg finds the view it will extract its schema (if TYPE=TABLE) into +a \s-1PG\s0 create table form, then it will extract the data (if TYPE=COPY or \s-1INSERT\s0) +following the view schema. +.PP +For example, with the following view: +.PP +.Vb 6 +\& CREATE OR REPLACE VIEW product_prices (category_id, product_count, low_price, high_price) AS +\& SELECT category_id, COUNT(*) as product_count, +\& MIN(list_price) as low_price, +\& MAX(list_price) as high_price +\& FROM product_information +\& GROUP BY category_id; +.Ve +.PP +Setting \s-1VIEW_AS_TABLE\s0 to product_prices and using export type \s-1TABLE,\s0 will +force Ora2Pg to detect columns returned types and to generate a create table +statement: +.PP +.Vb 6 +\& CREATE TABLE product_prices ( +\& category_id bigint, +\& product_count integer, +\& low_price numeric, +\& high_price numeric +\& ); +.Ve +.PP +Data will be loaded following the \s-1COPY\s0 or \s-1INSERT\s0 export type and the view +declaration. +.PP +You can use the \s-1ALLOW\s0 and \s-1EXCLUDE\s0 directive in addition to filter other +objects to export. +.SS "Export as Kettle transformation \s-1XML\s0 files" +.IX Subsection "Export as Kettle transformation XML files" +The \s-1KETTLE\s0 export type is useful if you want to use Penthalo Data Integrator +(Kettle) to import data to PostgreSQL. With this type of export Ora2Pg will +generate one \s-1XML\s0 Kettle transformation files (.ktr) per table and add a line +to manually execute the transformation in the output.sql file. For example: +.PP +.Vb 1 +\& ora2pg \-c ora2pg.conf \-t KETTLE \-j 12 \-a MYTABLE \-o load_mydata.sh +.Ve +.PP +will generate one file called '\s-1HR.MYTABLE\s0.ktr' and add a line to the output +file (load_mydata.sh): +.PP +.Vb 1 +\& #!/bin/sh +\& +\& KETTLE_TEMPLATE_PATH=\*(Aq.\*(Aq +\& +\& JAVAMAXMEM=4096 ./pan.sh \-file $KETTLE_TEMPLATE_PATH/HR.MYTABLE.ktr \-level Detailed +.Ve +.PP +The \-j 12 option will create a template with 12 processes to insert data into +PostgreSQL. It is also possible to specify the number of parallel queries used +to extract data from the Oracle with the \-J command line option as follow: +.PP +.Vb 1 +\& ora2pg \-c ora2pg.conf \-t KETTLE \-J 4 \-j 12 \-a EMPLOYEES \-o load_mydata.sh +.Ve +.PP +This is only possible if you have defined the technical key to used to split +the query between cores in the \s-1DEFINED_PKEY\s0 configuration directive. For example: +.PP +.Vb 1 +\& DEFINED_PK EMPLOYEES:employee_id +.Ve +.PP +will force the number of Oracle connection copies to 4 and defined the \s-1SQL\s0 query +as follow in the Kettle \s-1XML\s0 transformation file: +.PP +.Vb 1 +\& SELECT * FROM HR.EMPLOYEES WHERE ABS(MOD(employee_id,${Internal.Step.Unique.Count}))=${Internal.Step.Unique.Number} +.Ve +.PP +The \s-1KETTLE\s0 export type requires that the Oracle and PostgreSQL \s-1DSN\s0 are defined. +You can also activate the \s-1TRUNCATE_TABLE\s0 directive to force a truncation of the +table before data import. +.PP +The \s-1KETTLE\s0 export type is an original work of Marc Cousin. +.SS "Migration cost assessment" +.IX Subsection "Migration cost assessment" +Estimating the cost of a migration process from Oracle to PostgreSQL is not easy. To +obtain a good assessment of this migration cost, Ora2Pg will inspect all database +objects, all functions and stored procedures to detect if there's still some objects +and \s-1PL/SQL\s0 code that can not be automatically converted by Ora2Pg. +.PP +Ora2Pg has a content analysis mode that inspect the Oracle database to generate a +text report on what the Oracle database contains and what can not be exported. +.PP +To activate the \*(L"analysis and report\*(R" mode, you have to use the export de type +\&\s-1SHOW_REPORT\s0 like in the following command: +.PP +.Vb 1 +\& ora2pg \-t SHOW_REPORT +.Ve +.PP +Here is a sample report obtained with this command: +.PP +.Vb 6 +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& Ora2Pg: Oracle Database Content Report +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& Version Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 +\& Schema HR +\& Size 880.00 MB +\& +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& Object Number Invalid Comments +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& CLUSTER 2 0 Clusters are not supported and will not be exported. +\& FUNCTION 40 0 Total size of function code: 81992. +\& INDEX 435 0 232 index(es) are concerned by the export, others are automatically generated and will +\& do so on PostgreSQL. 1 bitmap index(es). 230 b\-tree index(es). 1 reversed b\-tree index(es) +\& Note that bitmap index(es) will be exported as b\-tree index(es) if any. Cluster, domain, +\& bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported +\& too, you may use a trigram\-based index (see pg_trgm) or a reverse() function based index +\& and search. You may also use \*(Aqvarchar_pattern_ops\*(Aq, \*(Aqtext_pattern_ops\*(Aq or \*(Aqbpchar_pattern_ops\*(Aq +\& operators in your indexes to improve search with the LIKE operator respectively into +\& varchar, text or char columns. +\& MATERIALIZED VIEW 1 0 All materialized view will be exported as snapshot materialized views, they +\& are only updated when fully refreshed. +\& PACKAGE BODY 2 1 Total size of package code: 20700. +\& PROCEDURE 7 0 Total size of procedure code: 19198. +\& SEQUENCE 160 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL +\& will be transformed into NEXTVAL(\*(Aqsequence_name\*(Aq) or CURRVAL(\*(Aqsequence_name\*(Aq). +\& TABLE 265 0 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration +\& directive to export as file_fdw foreign tables or use COPY in your code if you just +\& want to load data from external files. 2 binary columns. 4 unknown types. +\& TABLE PARTITION 8 0 Partitions are exported using table inheritance and check constraint. 1 HASH partitions. +\& 2 LIST partitions. 6 RANGE partitions. Note that Hash partitions are not supported. +\& TRIGGER 30 0 Total size of trigger code: 21677. +\& TYPE 7 1 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. +\& 2 Object type. 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type +\& inherited and Subtype are converted as table, type inheritance is not supported. +\& TYPE BODY 0 3 Export of type with member method are not supported, they will not be exported. +\& VIEW 7 0 Views are fully supported, but if you have updatable views you will need to use +\& INSTEAD OF triggers. +\& DATABASE LINK 1 0 Database links will not be exported. You may try the dblink perl contrib module or use +\& the SQL/MED PostgreSQL features with the different Foreign Data Wrapper (FDW) extensions. +\& +\& Note: Invalid code will not be exported unless the EXPORT_INVALID configuration directive is activated. +.Ve +.PP +Once the database can be analysed, Ora2Pg, by his ability to convert \s-1SQL\s0 and \s-1PL/SQL\s0 +code from Oracle syntax to PostgreSQL, can go further by estimating the code difficulties +and estimate the time necessary to operate a full database migration. +.PP +To estimate the migration cost in man-days, Ora2Pg allow you to use a configuration +directive called \s-1ESTIMATE_COST\s0 that you can also enabled at command line: +.PP +.Vb 1 +\& \-\-estimate_cost +.Ve +.PP +This feature can only be used with the \s-1SHOW_REPORT, FUNCTION, PROCEDURE, PACKAGE\s0 +and \s-1QUERY\s0 export type. +.PP +.Vb 1 +\& ora2pg \-t SHOW_REPORT \-\-estimate_cost +.Ve +.PP +The generated report is same as above but with a new 'Estimated cost' column as follow: +.PP +.Vb 6 +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& Ora2Pg: Oracle Database Content Report +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& Version Oracle Database 10g Express Edition Release 10.2.0.1.0 +\& Schema HR +\& Size 890.00 MB +\& +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& Object Number Invalid Estimated cost Comments +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& DATABASE LINK 3 0 9 Database links will be exported as SQL/MED PostgreSQL\*(Aqs Foreign Data Wrapper (FDW) extensions +\& using oracle_fdw. +\& FUNCTION 2 0 7 Total size of function code: 369 bytes. HIGH_SALARY: 2, VALIDATE_SSN: 3. +\& INDEX 21 0 11 11 index(es) are concerned by the export, others are automatically generated and will do so +\& on PostgreSQL. 11 b\-tree index(es). Note that bitmap index(es) will be exported as b\-tree +\& index(es) if any. Cluster, domain, bitmap join and IOT indexes will not be exported at all. +\& Reverse indexes are not exported too, you may use a trigram\-based index (see pg_trgm) or a +\& reverse() function based index and search. You may also use \*(Aqvarchar_pattern_ops\*(Aq, \*(Aqtext_pattern_ops\*(Aq +\& or \*(Aqbpchar_pattern_ops\*(Aq operators in your indexes to improve search with the LIKE operator +\& respectively into varchar, text or char columns. +\& JOB 0 0 0 Job are not exported. You may set external cron job with them. +\& MATERIALIZED VIEW 1 0 3 All materialized view will be exported as snapshot materialized views, they +\& are only updated when fully refreshed. +\& PACKAGE BODY 0 2 54 Total size of package code: 2487 bytes. Number of procedures and functions found +\& inside those packages: 7. two_proc.get_table: 10, emp_mgmt.create_dept: 4, +\& emp_mgmt.hire: 13, emp_mgmt.increase_comm: 4, emp_mgmt.increase_sal: 4, +\& emp_mgmt.remove_dept: 3, emp_mgmt.remove_emp: 2. +\& PROCEDURE 4 0 39 Total size of procedure code: 2436 bytes. TEST_COMMENTAIRE: 2, SECURE_DML: 3, +\& PHD_GET_TABLE: 24, ADD_JOB_HISTORY: 6. +\& SEQUENCE 3 0 0 Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL +\& will be transformed into NEXTVAL(\*(Aqsequence_name\*(Aq) or CURRVAL(\*(Aqsequence_name\*(Aq). +\& SYNONYM 3 0 4 SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround +\& is to use views or set the PostgreSQL search_path in your session to access +\& object outside the current schema. +\& user1.emp_details_view_v is an alias to hr.emp_details_view. +\& user1.emp_table is an alias to hr.employees@other_server. +\& user1.offices is an alias to hr.locations. +\& TABLE 17 0 8.5 1 external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration +\& directive to export as file_fdw foreign tables or use COPY in your code if you just want to +\& load data from external files. 2 binary columns. 4 unknown types. +\& TRIGGER 1 1 4 Total size of trigger code: 123 bytes. UPDATE_JOB_HISTORY: 2. +\& TYPE 7 1 5 5 type(s) are concerned by the export, others are not supported. 2 Nested Tables. 2 Object type. +\& 1 Subtype. 1 Type Boby. 1 Type inherited. 1 Varrays. Note that Type inherited and Subtype are +\& converted as table, type inheritance is not supported. +\& TYPE BODY 0 3 30 Export of type with member method are not supported, they will not be exported. +\& VIEW 1 1 1 Views are fully supported, but if you have updatable views you will need to use INSTEAD OF triggers. +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& Total 65 8 162.5 162.5 cost migration units means approximatively 2 man day(s). +.Ve +.PP +The last line shows the total estimated migration code in man-days following the +number of migration units estimated for each object. This migration unit represent +around five minutes for a PostgreSQL expert. If this is your first migration you can +get it higher with the configuration directive \s-1COST_UNIT_VALUE\s0 or the \-\-cost_unit_value +command line option: +.PP +.Vb 1 +\& ora2pg \-t SHOW_REPORT \-\-estimate_cost \-\-cost_unit_value 10 +.Ve +.PP +Ora2Pg is also able to give you a migration difficulty level assessment, here a sample: +.PP +Migration level: B\-5 +.PP +.Vb 10 +\& Migration levels: +\& A \- Migration that might be run automatically +\& B \- Migration with code rewrite and a human\-days cost up to 5 days +\& C \- Migration with code rewrite and a human\-days cost above 5 days +\& Technical levels: +\& 1 = trivial: no stored functions and no triggers +\& 2 = easy: no stored functions but with triggers, no manual rewriting +\& 3 = simple: stored functions and/or triggers, no manual rewriting +\& 4 = manual: no stored functions but with triggers or views with code rewriting +\& 5 = difficult: stored functions and/or triggers with code rewriting +.Ve +.PP +This assessment consist in a letter A or B to specify if the migration needs +manual rewriting or not. And a number from 1 up to 5 to give you a technical +difficulty level. You have an additional option \-\-human_days_limit to specify +the number of human-days limit where the migration level should be set to C +to indicate that it need a huge amount of work and a full project management +with migration support. Default is 10 human-days. You can use the configuration +directive \s-1HUMAN_DAYS_LIMIT\s0 to change this default value permanently. +.PP +This feature has been developed to help you or your boss to decide which +database to migrate first and the team that must be mobilized to operate +the migration. +.SS "Global Oracle and MySQL migration assessment" +.IX Subsection "Global Oracle and MySQL migration assessment" +Ora2Pg come with a script ora2pg_scanner that can be used when you have a huge +number of instances and schema to scan for migration assessment. +.PP +Usage: ora2pg_scanner \-l \s-1CSVFILE\s0 [\-o \s-1OUTDIR\s0] +.PP +.Vb 8 +\& \-b | \-\-binpath DIR: full path to directory where the ora2pg binary stays. +\& Might be useful only on Windows OS. +\& \-c | \-\-config FILE: set custom configuration file to use otherwise ora2pg +\& will use the default: /etc/ora2pg/ora2pg.conf. +\& \-l | \-\-list FILE : CSV file containing a list of databases to scan with +\& all required information. The first line of the file +\& can contain the following header that describes the +\& format that must be used: +\& +\& "type","schema/database","dsn","user","password" +\& +\& \-o | \-\-outdir DIR : (optional) by default all reports will be dumped to a +\& directory named \*(Aqoutput\*(Aq, it will be created automatically. +\& If you want to change the name of this directory, set the name +\& at second argument. +\& +\& \-t | \-\-test : just try all connections by retrieving the required schema +\& or database name. Useful to validate your CSV list file. +\& \-u | \-\-unit MIN : redefine globally the migration cost unit value in minutes. +\& Default is taken from the ora2pg.conf (default 5 minutes). +\& +\& Here is a full example of a CSV databases list file: +\& +\& "type","schema/database","dsn","user","password" +\& "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" +\& "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" +\& +\& The CSV field separator must be a comma. +\& +\& Note that if you want to scan all schemas from an Oracle instance you just +\& have to leave the schema field empty, Ora2Pg will automatically detect all +\& available schemas and generate a report for each one. Of course you need to +\& use a connection user with enough privileges to be able to scan all schemas. +\& For example: +\& +\& "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" +\& +\& will generate a report for all schema in the XE instance. Note that in this +\& case the SCHEMA directive in ora2pg.conf must not be set. +.Ve +.PP +It will generate a \s-1CSV\s0 file with the assessment result, one line per schema or +database and a detailed \s-1HTML\s0 report for each database scanned. +.PP +Hint: Use the \-t | \-\-test option before to test all your connections in your +\&\s-1CSV\s0 file. +.PP +For Windows users you must use the \-b command line option to set the directory +where ora2pg_scanner stays otherwise the ora2pg command calls will fail. +.PP +In the migration assessment details about functions Ora2Pg always include per +default 2 migration units for \s-1TEST\s0 and 1 unit for \s-1SIZE\s0 per 1000 characters in +the code. This mean that by default it will add 15 minutes in the migration +assessment per function. Obviously if you have unitary tests or very simple +functions this will not represent the real migration time. +.SS "Migration assessment method" +.IX Subsection "Migration assessment method" +Migration unit scores given to each type of Oracle database object are defined in the +Perl library lib/Ora2Pg/PLSQL.pm in the \f(CW%OBJECT_SCORE\fR variable definition. +.PP +The number of \s-1PL/SQL\s0 lines associated to a migration unit is also defined in this file +in the \f(CW$SIZE_SCORE\fR variable value. +.PP +The number of migration units associated to each \s-1PL/SQL\s0 code difficulties can be found +in the same Perl library lib/Ora2Pg/PLSQL.pm in the hash \f(CW%UNCOVERED_SCORE\fR initialization. +.PP +This assessment method is a work in progress so I'm expecting feedbacks on migration +experiences to polish the scores/units attributed in those variables. +.SS "Improving indexes and constraints creation speed" +.IX Subsection "Improving indexes and constraints creation speed" +Using the \s-1LOAD\s0 export type and a file containing \s-1SQL\s0 orders to perform, it is +possible to dispatch those orders over multiple PostgreSQL connections. To be +able to use this feature, the \s-1PG_DSN, PG_USER\s0 and \s-1PG_PWD\s0 must be set. Then: +.PP +.Vb 1 +\& ora2pg \-t LOAD \-c config/ora2pg.conf \-i schema/tables/INDEXES_table.sql \-j 4 +.Ve +.PP +will dispatch indexes creation over 4 simultaneous PostgreSQL connections. +.PP +This will considerably accelerate this part of the migration process with huge +data size. +.SS "Exporting \s-1LONG RAW\s0" +.IX Subsection "Exporting LONG RAW" +If you still have columns defined as \s-1LONG RAW,\s0 Ora2Pg will not be able to export +these kind of data. The \s-1OCI\s0 library fail to export them and always return the +same first record. To be able to export the data you need to transform the field +as \s-1BLOB\s0 by creating a temporary table before migrating data. For example, the +Oracle table: +.PP +.Vb 5 +\& SQL> DESC TEST_LONGRAW +\& Name NULL ? Type +\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- +\& ID NUMBER +\& C1 LONG RAW +.Ve +.PP +need to be \*(L"translated\*(R" into a table using \s-1BLOB\s0 as follow: +.PP +.Vb 1 +\& CREATE TABLE test_blob (id NUMBER, c1 BLOB); +.Ve +.PP +And then copy the data with the following \s-1INSERT\s0 query: +.PP +.Vb 1 +\& INSERT INTO test_blob SELECT id, to_lob(c1) FROM test_longraw; +.Ve +.PP +Then you just have to exclude the original table from the export (see \s-1EXCLUDE\s0 +directive) and to renamed the new temporary table on the fly using the +\&\s-1REPLACE_TABLES\s0 configuration directive. +.SS "Global variables" +.IX Subsection "Global variables" +Oracle allow the use of global variables defined in packages. Ora2Pg will +export these variables for PostgreSQL as user defined custom variables +available in a session. Oracle variables assignment are exported as +call to: +.PP +.Vb 1 +\& PERFORM set_config(\*(Aqpkgname.varname\*(Aq, value, false); +.Ve +.PP +Use of these variables in the code is replaced by: +.PP +.Vb 1 +\& current_setting(\*(Aqpkgname.varname\*(Aq)::global_variables_type; +.Ve +.PP +where global_variables_type is the type of the variable extracted from +the package definition. +.PP +If the variable is a constant or have a default value assigned at +declaration, Ora2Pg will create a file global_variables.conf with +the definition to include in the postgresql.conf file so that their +values will already be set at database connection. Note that the +value can always modified by the user so you can not have exactly +a constant. +.SS "Hints" +.IX Subsection "Hints" +Converting your queries with Oracle style outer join (+) syntax to \s-1ANSI\s0 standard \s-1SQL\s0 at +the Oracle side can save you lot of time for the migration. You can use \s-1TOAD\s0 Query Builder +can re-write these using the proper \s-1ANSI\s0 syntax, see: http://www.toadworld.com/products/toad\-for\-oracle/f/10/t/9518.aspx +.PP +There's also an alternative with \s-1SQL\s0 Developer Data Modeler, see +http://www.thatjeffsmith.com/archive/2012/01/sql\-developer\-data\-modeler\-quick\-tip\-use\-oracle\-join\-syntax\-or\-ansi/ +.PP +Toad is also able to rewrite the native Oracle \s-1\fBDECODE\s0()\fR syntax into \s-1ANSI\s0 +standard \s-1SQL CASE\s0 statement. You can find some slide about this in a +presentation given at PgConf.RU: http://ora2pg.darold.net/slides/ora2pg_the_hard_way.pdf +.SS "Test the migration" +.IX Subsection "Test the migration" +The type of action called \s-1TEST\s0 allow you to check that all objects from Oracle +database have been created under PostgreSQL. Of course \s-1PG_DSN\s0 must be set to be +able to check PostgreSQL side. +.PP +Note that this feature respect the schema set in the \s-1SCHEMA\s0 directive to scan +the Oracle database and also at PostgreSQL side if \s-1EXPORT_SCHEMA\s0 is enabled. +If \s-1PG_SCHEMA\s0 is defined and \s-1EXPORT_SCHEMA\s0 is enabled Ora2Pg will use the list +of schemas defined in \s-1PG_SCHEMA\s0 to scan PostgreSQL. If \s-1EXPORT_SCHEMA\s0 is +disabled the entire PostgreSQL database is scanned. +.PP +For example command: +.PP +.Vb 1 +\& ora2pg \-t TEST \-c config/ora2pg.conf > migration_diff.txt +.Ve +.PP +Will create a file containing the report of all object and row count on both +side, Oracle and PostgreSQL, with an error section giving you the detail of +the differences for each kind of object. Here is a sample result: +.PP +.Vb 10 +\& [TEST ROWS COUNT] +\& ORACLEDB:COUNTRIES:25 +\& POSTGRES:countries:25 +\& ORACLEDB:CUSTOMERS:6 +\& POSTGRES:customers:6 +\& ORACLEDB:DEPARTMENTS:27 +\& POSTGRES:departments:27 +\& ORACLEDB:EMPLOYEES:107 +\& POSTGRES:employees:107 +\& ORACLEDB:JOBS:19 +\& POSTGRES:jobs:19 +\& ORACLEDB:JOB_HISTORY:10 +\& POSTGRES:job_history:10 +\& ORACLEDB:LOCATIONS:23 +\& POSTGRES:locations:23 +\& ORACLEDB:PRODUCTS:0 +\& POSTGRES:products:0 +\& ORACLEDB:PTAB2:4 +\& ORACLEDB:REGIONS:4 +\& POSTGRES:regions:4 +\& [ERRORS ROWS COUNT] +\& Table ptab2 does not exists in PostgreSQL database. +\& +\& [TEST INDEXES COUNT] +\& ORACLEDB:COUNTRIES:1 +\& POSTGRES:countries:1 +\& ORACLEDB:JOB_HISTORY:4 +\& POSTGRES:job_history:4 +\& ORACLEDB:DEPARTMENTS:2 +\& POSTGRES:departments:1 +\& ORACLEDB:EMPLOYEES:6 +\& POSTGRES:employees:6 +\& ORACLEDB:CUSTOMERS:1 +\& POSTGRES:customers:1 +\& ORACLEDB:REGIONS:1 +\& POSTGRES:regions:1 +\& ORACLEDB:LOCATIONS:4 +\& POSTGRES:locations:4 +\& ORACLEDB:JOBS:1 +\& POSTGRES:jobs:1 +\& [ERRORS INDEXES COUNT] +\& Table departments doesn\*(Aqt have the same number of indexes in Oracle (2) and in PostgreSQL (1). +\& +\& [TEST VIEW COUNT] +\& ORACLEDB:VIEW:1 +\& POSTGRES:VIEW:1 +\& [ERRORS VIEW COUNT] +\& OK, Oracle and PostgreSQL have the same number of VIEW. +\& +\& [TEST MVIEW COUNT] +\& ORACLEDB:MVIEW:0 +\& POSTGRES:MVIEW:0 +\& [ERRORS MVIEW COUNT] +\& OK, Oracle and PostgreSQL have the same number of MVIEW. +\& +\& [TEST SEQUENCE COUNT] +\& ORACLEDB:SEQUENCE:1 +\& POSTGRES:SEQUENCE:0 +\& [ERRORS SEQUENCE COUNT] +\& SEQUENCE does not have the same count in Oracle (1) and in PostgreSQL (0). +\& +\& [TEST TYPE COUNT] +\& ORACLEDB:TYPE:1 +\& POSTGRES:TYPE:0 +\& [ERRORS TYPE COUNT] +\& TYPE does not have the same count in Oracle (1) and in PostgreSQL (0). +\& +\& [TEST FDW COUNT] +\& ORACLEDB:FDW:0 +\& POSTGRES:FDW:0 +\& [ERRORS FDW COUNT] +\& OK, Oracle and PostgreSQL have the same number of FDW. +.Ve +.PP +Here we can see that one table, one index, one sequence and one user defined +type have not been imported yet or have encountered an error. +.SH "SUPPORT" +.IX Header "SUPPORT" +.SS "Author / Maintainer" +.IX Subsection "Author / Maintainer" +Gilles Darold +.PP +Please report any bugs, patches, help, etc. to . +.SS "Feature request" +.IX Subsection "Feature request" +If you need new features let me know at . This help +a lot to develop a better/useful tool. +.SS "How to contribute ?" +.IX Subsection "How to contribute ?" +Any contribution to build a better tool is welcome, you just have to send me +your ideas, features request or patches and there will be applied. +.SH "LICENSE" +.IX Header "LICENSE" +Copyright (c) 2000\-2020 Gilles Darold \- All rights reserved. +.PP +.Vb 4 +\& This program is free software: you can redistribute it and/or modify +\& it under the terms of the GNU General Public License as published by +\& the Free Software Foundation, either version 3 of the License, or +\& any later version. +\& +\& This program is distributed in the hope that it will be useful, +\& but WITHOUT ANY WARRANTY; without even the implied warranty of +\& MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +\& GNU General Public License for more details. +\& +\& You should have received a copy of the GNU General Public License +\& along with this program. If not, see < http://www.gnu.org/licenses/ >. +.Ve +.SH "ACKNOWLEDGEMENT" +.IX Header "ACKNOWLEDGEMENT" +I must thanks a lot all the great contributors, see changelog for all acknowledgments. diff --git a/lib/Ora2Pg.pm b/lib/Ora2Pg.pm new file mode 100644 index 0000000000000000000000000000000000000000..a6b908f0f4c50d7ba0b017b447bab39b013f2972 --- /dev/null +++ b/lib/Ora2Pg.pm @@ -0,0 +1,20150 @@ +package Ora2Pg; +#------------------------------------------------------------------------------ +# Project : Oracle to PostgreSQL database schema converter +# Name : Ora2Pg.pm +# Language : Perl +# Authors : Gilles Darold, gilles _AT_ darold _DOT_ net +# Copyright: Copyright (c) 2000-2020 : Gilles Darold - All rights reserved - +# Function : Main module used to export Oracle database schema to PostgreSQL +# Usage : See documentation in this file with perldoc. +#------------------------------------------------------------------------------ +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see < http://www.gnu.org/licenses/ >. +# +#------------------------------------------------------------------------------ + +use vars qw($VERSION $PSQL %AConfig); +use Carp qw(confess); +use DBI; +use POSIX qw(locale_h _exit :sys_wait_h strftime); +use IO::File; +use Config; +use Time::HiRes qw/usleep/; +use Fcntl qw/ :flock /; +use IO::Handle; +use IO::Pipe; +use File::Basename; +use File::Spec qw/ tmpdir /; +use File::Temp qw/ tempfile /; +use Benchmark; + +#set locale to LC_NUMERIC C +setlocale(LC_NUMERIC,"C"); + +$VERSION = '21.1'; +$PSQL = $ENV{PLSQL} || 'psql'; + +$| = 1; + +our %RUNNING_PIDS = (); +# Multiprocess communication pipe +our $pipe = undef; +our $TMP_DIR = File::Spec->tmpdir() || '/tmp'; +our %ordered_views = (); + +# Character that must be escaped in COPY statement +my $ESCAPE_COPY = { "\0" => "", "\\" => "\\\\", "\r" => "\\r", "\n" => "\\n", "\t" => "\\t"}; + +# Oracle internal timestamp month equivalent +our %ORACLE_MONTHS = ('JAN'=>'01', 'FEB'=>'02','MAR'=>'03','APR'=>'04','MAY'=>'05','JUN'=>'06','JUL'=>'07','AUG'=>'08','SEP'=>'09','OCT'=>10,'NOV'=>11,'DEC'=>12); + +# Exclude table generated by partition logging, materialized view logs, statistis on spatial index, +# spatial index tables, sequence index tables, interMedia Text index tables and Unified Audit tables. +# LogMiner, Oracle Advanced Replication, hash table used by loadjava. +our @EXCLUDED_TABLES = ('USLOG\$_.*', 'MLOG\$_.*', 'RUPD\$_.*', 'MDXT_.*', 'MDRT_.*', 'MDRS_.*', 'DR\$.*', 'CLI_SWP\$.*', 'LOGMNR\$.*', 'REPCAT\$.*', 'JAVA\$.*', 'AQ\$.*', 'BIN\$.*', 'SDO_GR_.*', '.*\$JAVA\$.*', 'PROF\$.*', 'TOAD_PLAN_.*', 'SYS_.*\$', 'QUEST_SL_.*'); +our @EXCLUDED_TABLES_8I = ('USLOG$_%', 'MLOG$_%', 'RUPD$_%', 'MDXT_%', 'MDRT_%', 'MDRS_%', 'DR$%', 'CLI_SWP$%', 'LOGMNR$%', 'REPCAT$%', 'JAVA$%', 'AQ$%', 'BIN$%', '%$JAVA$%', 'PROF$%', 'TOAD_PLAN_%', 'SYS_%$', 'QUEST_SL_%'); + +our @Oracle_tables = qw( +EVT_CARRIER_CONFIGURATION +EVT_DEST_PROFILE +EVT_HISTORY +EVT_INSTANCE +EVT_MAIL_CONFIGURATION +EVT_MONITOR_NODE +EVT_NOTIFY_STATUS +EVT_OPERATORS +EVT_OPERATORS_ADDITIONAL +EVT_OPERATORS_SYSTEMS +EVT_OUTSTANDING +EVT_PROFILE +EVT_PROFILE_EVENTS +EVT_REGISTRY +EVT_REGISTRY_BACKLOG +OLS_DIR_BUSINESSE +OLS_DIR_BUSINESSES +SDO_COORD_REF_SYS +SDO_CS_SRS +SDO_INDEX_METADATA_TABLE +SDO_INDEX_METADATA_TABLES +SDO_PC_BLK_TABLE +SDO_STYLES_TABLE +SDO_TIN_BLK_TABLE +SMACTUALPARAMETER_S +SMGLOBALCONFIGURATION_S +SMFORMALPARAMETER_S +SMFOLDER_S +SMDISTRIBUTIONSET_S +SMDEPENDENTLINKS +SMDEPENDENTINDEX +SMDEPENDEEINDEX +SMDEFAUTH_S +SMDBAUTH_S +SMPARALLELJOB_S +SMPACKAGE_S +SMOWNERLINKS +SMOWNERINDEX +SMOWNEEINDEX +SMOSNAMES_X +SMOMSTRING_S +SMMOWNERLINKS +SMMOWNERINDEX +SMPACKAGE_S +SMPARALLELJOB_S +SMPARALLELOPERATION_S +SMPARALLELSTATEMENT_S +SMPRODUCT_S +SMP_AD_ADDRESSES_ +SMP_AD_DISCOVERED_NODES_ +SMP_AD_NODES_ +SMP_AD_PARMS_ +SMP_AUTO_DISCOVERY_ITEM_ +SMP_AUTO_DISCOVERY_PARMS_ +SMP_BLOB_ +SMP_CREDENTIALS\$ +SMP_JOB_ +SMP_JOB_EVENTLIST_ +SMP_JOB_HISTORY_ +SMP_JOB_INSTANCE_ +SMP_JOB_LIBRARY_ +SMP_JOB_TASK_INSTANCE_ +SMP_LONG_TEXT_ +SMP_REP_VERSION +SMP_SERVICES +SMP_SERVICE_GROUP_DEFN_ +SMP_SERVICE_GROUP_ITEM_ +SMP_SERVICE_ITEM_ +SMP_UPDATESERVICES_CALLED_ +SMAGENTJOB_S +SMARCHIVE_S +SMBREAKABLELINKS +SMCLIQUE +SMCONFIGURATION +SMCONSOLESOSETTING_S +SMDATABASE_S +SMHOSTAUTH_S +SMHOST_S +SMINSTALLATION_S +SMLOGMESSAGE_S +SMMONTHLYENTRY_S +SMMONTHWEEKENTRY_S +SMP_USER_DETAILS +SMRELEASE_S +SMRUN_S +SMSCHEDULE_S +SMSHAREDORACLECLIENT_S +SMSHAREDORACLECONFIGURATION_S +SMTABLESPACE_S +SMVCENDPOINT_S +SMWEEKLYENTRY_S +); +push(@EXCLUDED_TABLES, @Oracle_tables); + +# Some function might be excluded from export and assessment. +our @EXCLUDED_FUNCTION = ('SQUIRREL_GET_ERROR_OFFSET'); + +our @FKEY_OPTIONS = ('NEVER', 'DELETE', 'ALWAYS'); + +# Minimized the footprint on disc, so that more rows fit on a data page, +# which is the most important factor for speed. +our %TYPALIGN = ( + # Types and size, 1000 = variable + 'boolean' => 1, + 'smallint' => 2, + 'smallserial' => 2, + 'integer' => 4, + 'real' => 4, + 'serial' => 4, + 'date' => 4, + 'oid' => 4, + 'macaddr' => 6, + 'bigint' => 8, + 'bigserial' => 8, + 'double precision' => 8, + 'macaddr8' => 8, + 'money' => 8, + 'time' => 8, + 'timestamp' => 8, + 'timestamp without time zone' => 8, + 'timestamp with time zone' => 8, + 'interval' => 16, + 'point' => 16, + 'tinterval' => 16, + 'uuid' => 16, + 'circle' => 24, + 'box' => 32, + 'line' => 32, + 'lseg' => 32, + 'bit' => 1000, + 'bytea' => 1000, + 'character varying' => 1000, + 'cidr' => 19, + 'json' => 1000, + 'jsonb' => 1000, + 'numeric' => 1000, + 'path' => 1000, + 'polygon' => 1000, + 'text' => 1000, + 'xml' => 1000, + # aliases + 'bool' => 1, + 'timetz' => 12, + 'char' => 1000, + 'decimal' => 1000, + # deprecated + 'int2' => 2, + 'abstime' => 4, + 'bpchar' => 4, + 'int4' => 4, + 'reltime' => 4, + 'float4' => 4, + 'timestamptz' => 8, + 'float8' => 8, + 'int8' => 8, + 'name' => 64, + 'inet' => 19, + 'varbit' => 1000, + 'varchar' => 1000 +); + +# These definitions can be overriden from configuration file +our %TYPE = ( + # Oracle only has one flexible underlying numeric type, NUMBER. + # Without precision and scale it is set to the PG type float8 + # to match all needs + 'NUMBER' => 'numeric', + # CHAR types limit of 2000 bytes with defaults to 1 if no length + # is specified. PG char type has max length set to 8104 so it + # should match all needs + 'CHAR' => 'char', + 'NCHAR' => 'char', + # VARCHAR types the limit is 2000 bytes in Oracle 7 and 4000 in + # Oracle 8. PG varchar type has max length iset to 8104 so it + # should match all needs + 'VARCHAR' => 'varchar', + 'NVARCHAR' => 'varchar', + 'VARCHAR2' => 'varchar', + 'NVARCHAR2' => 'varchar', + 'STRING' => 'varchar', + # The DATE data type is used to store the date and time + # information. PG type timestamp should match all needs. + 'DATE' => 'timestamp', + # Type LONG is like VARCHAR2 but with up to 2Gb. PG type text + # should match all needs or if you want you could use blob + 'LONG' => 'text', # Character data of variable length + 'LONG RAW' => 'bytea', + # Types LOB and FILE are like LONG but with up to 4Gb. PG type + # text should match all needs or if you want you could use blob + # (large object) + 'CLOB' => 'text', # A large object containing single-byte characters + 'NCLOB' => 'text', # A large object containing national character set data + 'BLOB' => 'bytea', # Binary large object + # The full path to the external file is returned if destination type is text. + # If the destination type is bytea the content of the external file is returned. + 'BFILE' => 'bytea', # Locator for external large binary file + # The RAW type is presented as hexadecimal characters. The + # contents are treated as binary data. Limit of 2000 bytes + # PG type text should match all needs or if you want you could + # use blob (large object) + 'RAW' => 'bytea', + 'ROWID' => 'oid', + 'UROWID' => 'oid', + 'FLOAT' => 'double precision', + 'DEC' => 'decimal', + 'DECIMAL' => 'decimal', + 'DOUBLE PRECISION' => 'double precision', + 'INT' => 'numeric', + 'INTEGER' => 'numeric', + 'BINARY_INTEGER' => 'integer', + 'PLS_INTEGER' => 'integer', + 'REAL' => 'real', + 'SMALLINT' => 'smallint', + 'BINARY_FLOAT' => 'double precision', + 'BINARY_DOUBLE' => 'double precision', + 'TIMESTAMP' => 'timestamp', + 'BOOLEAN' => 'boolean', + 'INTERVAL' => 'interval', + 'XMLTYPE' => 'xml', + 'TIMESTAMP WITH TIME ZONE' => 'timestamp with time zone', + 'TIMESTAMP WITH LOCAL TIME ZONE' => 'timestamp with time zone', + 'SDO_GEOMETRY' => 'geometry', +); + +our %ORA2PG_SDO_GTYPE = ( + '0' => 'GEOMETRY', + '1' => 'POINT', + '2' => 'LINESTRING', + '3' => 'POLYGON', + '4' => 'GEOMETRYCOLLECTION', + '5' => 'MULTIPOINT', + '6' => 'MULTILINESTRING', + '7' => 'MULTIPOLYGON', + '8' => 'SOLID', + '9' => 'MULTISOLID' +); + +our %GTYPE = ( + 'UNKNOWN_GEOMETRY' => 'GEOMETRY', + 'GEOMETRY' => 'GEOMETRY', + 'POINT' => 'POINT', + 'LINE' => 'LINESTRING', + 'CURVE' => 'LINESTRING', + 'POLYGON' => 'POLYGON', + 'SURFACE' => 'POLYGON', + 'COLLECTION' => 'GEOMETRYCOLLECTION', + 'MULTIPOINT' => 'MULTIPOINT', + 'MULTILINE' => 'MULTILINESTRING', + 'MULTICURVE' => 'MULTILINESTRING', + 'MULTIPOLYGON' => 'MULTIPOLYGON', + 'MULTISURFACE' => 'MULTIPOLYGON', + 'SOLID' => 'SOLID', + 'MULTISOLID' => 'MULTISOLID' +); +our %INDEX_TYPE = ( + 'NORMAL' => 'b-tree', + 'NORMAL/REV' => 'reversed b-tree', + 'FUNCTION-BASED NORMAL' => 'function based b-tree', + 'FUNCTION-BASED NORMAL/REV' => 'function based reversed b-tree', + 'BITMAP' => 'bitmap', + 'BITMAP JOIN' => 'bitmap join', + 'FUNCTION-BASED BITMAP' => 'function based bitmap', + 'FUNCTION-BASED BITMAP JOIN' => 'function based bitmap join', + 'CLUSTER' => 'cluster', + 'DOMAIN' => 'domain', + 'IOT - TOP' => 'IOT', + 'SPATIAL INDEX' => 'spatial index', +); + +# Reserved keywords in PostgreSQL +our @KEYWORDS = qw( + ALL ANALYSE ANALYZE AND ANY ARRAY AS ASC ASYMMETRIC AUTHORIZATION BINARY + BOTH CASE CAST CHECK COLLATE COLLATION COLUMN CONCURRENTLY CONSTRAINT CREATE + CROSS CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME + CURRENT_TIMESTAMP CURRENT_USER DEFAULT DEFERRABLE DESC DISTINCT DO ELSE END + EXCEPT FALSE FETCH FOR FOREIGN FREEZE FROM FULL GRANT GROUP HAVING ILIKE IN + INITIALLY INNER INTERSECT INTO IS ISNULL JOIN LATERAL LEADING LEFT LIKE LIMIT + LOCALTIME LOCALTIMESTAMP NATURAL NOT NOTNULL NULL OFFSET ON ONLY OR ORDER OUTER + OVERLAPS PLACING PRIMARY REFERENCES RETURNING RIGHT SELECT SESSION_USER SIMILAR + SOME SYMMETRIC TABLE TABLESAMPLE THEN TO TRAILING TRUE UNION UNIQUE USER USING + VARIADIC VERBOSE WHEN WHERE WINDOW WITH +); + +# Reserved keywords that can be used in PostgreSQL as function or type name +our @FCT_TYPE_KEYWORDS = qw( + AUTHORIZATION BINARY COLLATION CONCURRENTLY CROSS CURRENT_SCHEMA FREEZE + FULL ILIKE INNER IS ISNULL JOIN LEFT LIKE NATURAL NOTNULL OUTER OVERLAPS + RIGHT SIMILAR TABLESAMPLE VERBOSE +); + + +our @SYSTEM_FIELDS = qw(oid tableoid xmin xmin cmin xmax cmax ctid); +our %BOOLEAN_MAP = ( + 'yes' => 't', + 'no' => 'f', + 'y' => 't', + 'n' => 'f', + '1' => 't', + '0' => 'f', + 'true' => 't', + 'false' => 'f', + 'enabled'=> 't', + 'disabled'=> 'f', + 't' => 't', + 'f' => 'f', +); + +our @GRANTS = ( + 'SELECT', 'INSERT', 'UPDATE', 'DELETE', 'TRUNCATE', + 'REFERENCES', 'TRIGGER', 'USAGE', 'CREATE', 'CONNECT', + 'TEMPORARY', 'TEMP', 'USAGE', 'ALL', 'ALL PRIVILEGES', + 'EXECUTE' +); + +$SIG{'CHLD'} = 'DEFAULT'; + +#### +# method used to fork as many child as wanted +## +sub spawn +{ + my $coderef = shift; + + unless (@_ == 0 && $coderef && ref($coderef) eq 'CODE') { + print "usage: spawn CODEREF"; + exit 0; + } + + my $pid; + if (!defined($pid = fork)) { + print STDERR "Error: cannot fork: $!\n"; + return; + } elsif ($pid) { + $RUNNING_PIDS{$pid} = $pid; + return; # the parent + } + # the child -- go spawn + $< = $>; + $( = $); # suid progs only + exit &$coderef(); +} + +# With multiprocess we need to wait all childs +sub wait_child +{ + my $sig = shift; + print STDERR "Received terminating signal ($sig).\n"; + if ($^O !~ /MSWin32|dos/i) { + 1 while wait != -1; + $SIG{INT} = \&wait_child; + $SIG{TERM} = \&wait_child; + } + print STDERR "Aborting.\n"; + _exit(0); +} +$SIG{INT} = \&wait_child; +$SIG{TERM} = \&wait_child; + +=head1 PUBLIC METHODS + +=head2 new HASH_OPTIONS + +Creates a new Ora2Pg object. + +The only required option is: + + - config : Path to the configuration file (required). + +All directives found in the configuration file can be overwritten in the +instance call by passing them in lowercase as arguments. + +=cut + +sub new +{ + my ($class, %options) = @_; + + # This create an OO perl object + my $self = {}; + bless ($self, $class); + + # Initialize this object + $self->_init(%options); + + # Return the instance + return($self); +} + + + +=head2 export_schema FILENAME + +Print SQL data output to a file name or +to STDOUT if no file name is specified. + +=cut + +sub export_schema +{ + my $self = shift; + + # Create default export file where things will be written with the dump() method + # First remove it if the output file already exists + foreach my $t (@{$self->{export_type}}) + { + next if ($t =~ /^(?:SHOW_|TEST)/i); # SHOW_* commands are not concerned here + + # Set current export type + $self->{type} = $t; + + if ($self->{type} ne 'LOAD') + { + # Close open main output file + if (defined $self->{fhout}) { + $self->close_export_file($self->{fhout}); + } + # Remove old export file if it already exists + $self->remove_export_file(); + # then create a new one + $self->create_export_file(); + } + + # Dump exported statement to output + $self->_get_sql_statements(); + + if ($self->{type} ne 'LOAD') + { + # Close output export file create above + $self->close_export_file($self->{fhout}) if (defined $self->{fhout}); + } + } + + # Disconnect from the database + $self->{dbh}->disconnect() if ($self->{dbh}); + $self->{dbhdest}->disconnect() if ($self->{dbhdest}); + + # Try to requalify package function call + if (!$self->{package_as_schema}) { + $self->fix_function_call(); + } + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + unlink($dirprefix . 'temp_pass2_file.dat'); +} + + +=head2 open_export_file FILENAME + +Open a file handle to a given filename. + +=cut + +sub open_export_file +{ + my ($self, $outfile, $noprefix) = @_; + + my $filehdl = undef; + + if ($outfile && $outfile ne '-') { + if ($outfile ne '-') { + if ($self->{output_dir} && !$noprefix) { + $outfile = $self->{output_dir} . '/' . $outfile; + } + if ($self->{input_file} && ($outfile eq $self->{input_file})) { + $self->logit("FATAL: input file is the same as output file: $outfile, can not overwrite it.\n",0,1); + } + } + # If user request data compression + if ($outfile =~ /\.gz$/i) { + eval("use Compress::Zlib;"); + $self->{compress} = 'Zlib'; + $filehdl = gzopen("$outfile", "wb") or $self->logit("FATAL: Can't create deflation file $outfile\n",0,1); + } elsif ($outfile =~ /\.bz2$/i) { + $self->logit("Error: can't run bzip2\n",0,1) if (!-x $self->{bzip2}); + $self->{compress} = 'Bzip2'; + $filehdl = new IO::File; + $filehdl->open("|$self->{bzip2} --stdout >$outfile") or $self->logit("FATAL: Can't open pipe to $self->{bzip2} --stdout >$outfile: $!\n", 0,1); + } else { + $filehdl = new IO::File; + $filehdl->open(">$outfile") or $self->logit("FATAL: Can't open $outfile: $!\n", 0, 1); + } + $filehdl->autoflush(1) if (defined $filehdl && !$self->{compress}); + } + + return $filehdl; +} + +=head2 create_export_file FILENAME + +Set output file and open a file handle on it, +will use STDOUT if no file name is specified. + +=cut + +sub create_export_file +{ + my ($self, $outfile) = @_; + + # Do not create the default export file with direct data export + if (($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY')) { + return if ($self->{pg_dsn}); + } + + # Init with configuration OUTPUT filename + $outfile ||= $self->{output}; + if ($outfile) + { + if ($outfile ne '-') + { + # Prefix out file with export type in multiple export type call + $outfile = $self->{type} . "_$outfile" if ($#{$self->{export_type}} > 0); + if ($self->{output_dir} && $outfile) { + $outfile = $self->{output_dir} . "/" . $outfile; + } + if ($self->{input_file} && ($outfile eq $self->{input_file})) { + $self->logit("FATAL: input file is the same as output file: $outfile, can not overwrite it.\n",0,1); + } + } + + # Send output to the specified file + if ($outfile =~ /\.gz$/) + { + eval("use Compress::Zlib;"); + $self->{compress} = 'Zlib'; + $self->{fhout} = gzopen($outfile, "wb") or $self->logit("FATAL: Can't create deflation file $outfile\n", 0, 1); + } + elsif ($outfile =~ /\.bz2$/) + { + $self->logit("FATAL: can't run bzip2\n",0,1) if (!-x $self->{bzip2}); + $self->{compress} = 'Bzip2'; + $self->{fhout} = new IO::File; + $self->{fhout}->open("|$self->{bzip2} --stdout >$outfile") or $self->logit("FATAL: Can't open pipe to $self->{bzip2} --stdout >$outfile: $!\n", 0, 1); + } + else + { + $self->{fhout} = new IO::File; + $self->{fhout}->open(">>$outfile") or $self->logit("FATAL: Can't open $outfile: $!\n", 0, 1); + $self->set_binmode($self->{fhout}); + } + if ( $self->{compress} && (($self->{jobs} > 1) || ($self->{oracle_copies} > 1)) ) + { + die "FATAL: you can't use compressed output with parallel dump\n"; + } + } +} + +sub remove_export_file +{ + my ($self, $outfile) = @_; + + # Init with configuration OUTPUT filename + $outfile ||= $self->{output}; + if ($outfile && $outfile ne '-') + { + # Prefix out file with export type in multiple export type call + $outfile = $self->{type} . "_$outfile" if ($#{$self->{export_type}} > 0); + if ($self->{output_dir} && $outfile) + { + $outfile = $self->{output_dir} . "/" . $outfile; + } + if ($self->{input_file} && ($outfile eq $self->{input_file})) + { + $self->logit("FATAL: input file is the same as output file: $outfile, can not overwrite it.\n",0,1); + } + unlink($outfile); + } +} + +=head2 append_export_file FILENAME + +Open a file handle to a given filename to append data. + +=cut + +sub append_export_file +{ + my ($self, $outfile, $noprefix) = @_; + + my $filehdl = undef; + + if ($outfile) + { + if ($self->{output_dir} && !$noprefix) { + $outfile = $self->{output_dir} . '/' . $outfile; + } + # If user request data compression + if ($self->{compress} && (($self->{jobs} > 1) || ($self->{oracle_copies} > 1))) { + die "FATAL: you can't use compressed output with parallel dump\n"; + } else { + $filehdl = new IO::File; + $filehdl->open(">>$outfile") or $self->logit("FATAL: Can't open $outfile: $!\n", 0, 1); + $filehdl->autoflush(1); + } + } + + return $filehdl; +} + +=head2 read_export_file FILENAME + +Open a file handle to a given filename to read data. + +=cut + +sub read_export_file +{ + my ($self, $infile) = @_; + + my $filehdl = new IO::File; + $filehdl->open("<$infile") or $self->logit("FATAL: Can't read $infile: $!\n", 0, 1); + + return $filehdl; +} + + +=head2 close_export_file FILEHANDLE + +Close a file handle. + +=cut + +sub close_export_file +{ + my ($self, $filehdl, $not_compressed) = @_; + + + return if (!defined $filehdl); + + if (!$not_compressed && $self->{output} =~ /\.gz$/) { + $filehdl->gzclose(); + } else { + $filehdl->close(); + } +} + +=head2 modify_struct TABLE_NAME ARRAYOF_FIELDNAME + +Modify the table structure during the export. Only the specified columns +will be exported. + +=cut + +sub modify_struct +{ + my ($self, $table, @fields) = @_; + + if (!$self->{preserve_case}) { + map { $_ = lc($_) } @fields; + $table = lc($table); + } + push(@{$self->{modify}{$table}}, @fields); + +} + +=head2 is_reserved_words + +Returns 1 if the given object name is a PostgreSQL reserved word +Returns 2 if the object name is only numeric +Returns 3 if the object name is a system column + +=cut + +sub is_reserved_words +{ + my ($self, $obj_name) = @_; + + if ($obj_name && grep(/^\Q$obj_name\E$/i, @KEYWORDS)) { + return 1 if (!grep(/^$self->{type}/, 'FUNCTION', 'PACKAGE', 'PROCEDURE') || grep(/^\Q$obj_name\E$/i, @FCT_TYPE_KEYWORDS)); + } + if ($obj_name =~ /^\d+/) { + return 2; + } + if ($obj_name && grep(/^\Q$obj_name\E$/i, @SYSTEM_FIELDS)) { + return 3; + } + + return 0; +} + +=head2 quote_object_name + +Return a quoted object named when needed: + - PostgreSQL reserved word + - unsupported character + - start with a digit or digit only +=cut + + +sub quote_object_name +{ + my ($self, @obj_list) = @_; + + my @ret = (); + + foreach my $obj_name (@obj_list) + { + next if ($obj_name =~ /^SYS_NC\d+/); + + # Start by removing any double quote and extra space + $obj_name =~ s/"//g; + $obj_name =~ s/^\s+//; + $obj_name =~ s/\s+$//; + + # When PRESERVE_CASE is not enabled set object name to lower case + if (!$self->{preserve_case}) + { + $obj_name = lc($obj_name); + # then if there is non alphanumeric or the object name is a reserved word + if ($obj_name =~ /[^a-z0-9\_\.]/ || ($self->{use_reserved_words} && $self->is_reserved_words($obj_name)) || $obj_name =~ /^\d+/) + { + # Add double quote to [schema.] object name + if ($obj_name !~ /^[^\.]+\.[^\.]+$/ && $obj_name !~ /^[^\.]+\.[^\.]+\.[^\.]+$/) { + $obj_name = '"' . $obj_name . '"'; + } elsif ($obj_name =~ /^[^\.]+\.[^\.]+$/) { + $obj_name =~ s/^([^\.]+)\.([^\.]+)$/"$1"\."$2"/; + } else { + $obj_name =~ s/^([^\.]+)\.([^\.]+)\.([^\.]+)$/"$1"\."$2"\."$3"/; + } + } + } + # Add double quote to [schema.] object name + elsif ($obj_name !~ /^[^\.]+\.[^\.]+$/ && $obj_name !~ /^[^\.]+\.[^\.]+\.[^\.]+$/) { + $obj_name = "\"$obj_name\""; + } elsif ($obj_name =~ /^[^\.]+\.[^\.]+$/) { + $obj_name =~ s/^([^\.]+)\.([^\.]+)$/"$1"\."$2"/; + } else { + $obj_name =~ s/^([^\.]+)\.([^\.]+)\.([^\.]+)$/"$1"\."$2"\."$3"/; + } + push(@ret, $obj_name); + } + + return join(',', @ret); +} + +=head2 replace_tables HASH + +Modify table names during the export. + +=cut + +sub replace_tables +{ + my ($self, %tables) = @_; + + foreach my $t (keys %tables) { + $self->{replaced_tables}{"\L$t\E"} = $tables{$t}; + } + +} + +=head2 replace_cols HASH + +Modify column names during the export. + +=cut + +sub replace_cols +{ + my ($self, %cols) = @_; + + foreach my $t (keys %cols) { + foreach my $c (keys %{$cols{$t}}) { + $self->{replaced_cols}{"\L$t\E"}{"\L$c\E"} = $cols{$t}{$c}; + } + } + +} + +=head2 set_where_clause HASH + +Add a WHERE clause during data export on specific tables or on all tables + +=cut + +sub set_where_clause +{ + my ($self, $global, %table_clause) = @_; + + $self->{global_where} = $global; + foreach my $t (keys %table_clause) { + $self->{where}{"\L$t\E"} = $table_clause{$t}; + } + +} + +=head2 set_delete_clause HASH + +Add a DELETE clause before data export on specific tables or on all tables + +=cut + +sub set_delete_clause +{ + my ($self, $global, %table_clause) = @_; + + $self->{global_delete} = $global; + foreach my $t (keys %table_clause) { + $self->{delete}{"\L$t\E"} = $table_clause{$t}; + } + +} + + +#### Private subroutines #### + +=head1 PRIVATE METHODS + +=head2 _init HASH_OPTIONS + +Initialize an Ora2Pg object instance with a connexion to the +Oracle database. + +=cut + +sub _init +{ + my ($self, %options) = @_; + + # Use custom temp directory if specified + $TMP_DIR = $options{temp_dir} || $TMP_DIR; + + # Read configuration file + $self->read_config($options{config}) if ($options{config}); + + # Those are needed by DBI + $ENV{ORACLE_HOME} = $AConfig{'ORACLE_HOME'} if ($AConfig{'ORACLE_HOME'}); + $ENV{NLS_LANG} = $AConfig{'NLS_LANG'} if ($AConfig{'NLS_LANG'}); + + # Init arrays + $self->{default_tablespaces} = (); + $self->{limited} = (); + $self->{excluded} = (); + $self->{view_as_table} = (); + $self->{modify} = (); + $self->{replaced_tables} = (); + $self->{replaced_cols} = (); + $self->{replace_as_boolean} = (); + $self->{ora_boolean_values} = (); + $self->{null_equal_empty} = 1; + $self->{estimate_cost} = 0; + $self->{where} = (); + $self->{replace_query} = (); + $self->{ora_reserved_words} = (); + $self->{defined_pk} = (); + $self->{allow_partition} = (); + $self->{empty_lob_null} = 0; + $self->{look_forward_function} = (); + $self->{no_function_metadata} = 0; + + # Initial command to execute at Oracle and PostgreSQL connexion + $self->{ora_initial_command} = (); + $self->{pg_initial_command} = (); + + # To register user defined exception + $self->{custom_exception} = (); + $self->{exception_id} = 50001; + + # Init PostgreSQL DB handle + $self->{dbhdest} = undef; + $self->{standard_conforming_strings} = 1; + $self->{create_schema} = 1; + + # Init some arrays + $self->{external_table} = (); + $self->{function_metadata} = (); + $self->{grant_object} = ''; + + # Used to precise if we need to prefix partition tablename with main tablename + $self->{prefix_partition} = 0; + $self->{prefix_part_subpartition} = 1; + + # Use to preserve the data export type with geometry objects + $self->{local_type} = ''; + + # Shall we log on error during data import or abort. + $self->{log_on_error} = 0; + + # Initialize some variable related to export of mysql database + $self->{is_mysql} = 0; + $self->{mysql_mode} = ''; + $self->{mysql_internal_extract_format} = 0; + $self->{mysql_pipes_as_concat} = 0; + + # List of users for audit trail + $self->{audit_user} = ''; + + # Disable copy freeze by default + $self->{copy_freeze} = ''; + + # Use FTS index to convert CONTEXT Oracle's indexes by default + $self->{context_as_trgm} = 0; + $self->{fts_index_only} = 1; + $self->{fts_config} = ''; + $self->{use_unaccent} = 1; + $self->{use_lower_unaccent} = 1; + + # Enable rewrite of outer join by default. + $self->{rewrite_outer_join} = 1; + + # Init comment and text constant storage variables + $self->{idxcomment} = 0; + $self->{comment_values} = (); + $self->{text_values} = (); + $self->{text_values_pos} = 0; + + # Keep commit/rollback in converted pl/sql code by default + $self->{comment_commit_rollback} = 0; + + # Keep savepoint in converted pl/sql code by default + $self->{comment_savepoint} = 0; + + # Storage of string constant placeholder regexp + $self->{string_constant_regexp} = (); + $self->{alternative_quoting_regexp} = (); + + # Global file handle + $self->{cfhout} = undef; + + # Initialyze following configuration file + foreach my $k (sort keys %AConfig) { + if (lc($k) eq 'allow') { + $self->{limited} = $AConfig{ALLOW}; + } elsif (lc($k) eq 'exclude') { + $self->{excluded} = $AConfig{EXCLUDE}; + } else { + $self->{lc($k)} = $AConfig{$k}; + } + } + + # Set default system user/schema to not export. + push(@{$self->{sysusers}},'SYSTEM','CTXSYS','DBSNMP','EXFSYS','LBACSYS','MDSYS','MGMT_VIEW','OLAPSYS','ORDDATA','OWBSYS','ORDPLUGINS','ORDSYS','OUTLN','SI_INFORMTN_SCHEMA','SYS','SYSMAN','WK_TEST','WKSYS','WKPROXY','WMSYS','XDB','APEX_PUBLIC_USER','DIP','FLOWS_020100','FLOWS_030000','FLOWS_040100','FLOWS_010600','FLOWS_FILES','MDDATA','ORACLE_OCM','SPATIAL_CSW_ADMIN_USR','SPATIAL_WFS_ADMIN_USR','XS$NULL','PERFSTAT','SQLTXPLAIN','DMSYS','TSMSYS','WKSYS','APEX_040000','APEX_040200','DVSYS','OJVMSYS','GSMADMIN_INTERNAL','APPQOSSYS','DVSYS','DVF','AUDSYS','APEX_030200','MGMT_VIEW','ODM','ODM_MTR','TRACESRV','MTMSYS','OWBSYS_AUDIT','WEBSYS','WK_PROXY','OSE$HTTP$ADMIN','AURORA$JIS$UTILITY$','AURORA$ORB$UNAUTHENTICATED','DBMS_PRIVILEGE_CAPTURE','CSMIG', 'MGDSYS', 'SDE','DBSFWUSER'); + + # Set default tablespace to exclude when using USE_TABLESPACE + push(@{$self->{default_tablespaces}}, 'TEMP', 'USERS','SYSTEM'); + + # Verify grant objects + if ($self->{type} eq 'GRANT' && $self->{grant_object}) { + die "FATAL: wrong object type in GRANT_OBJECTS directive.\n" if (!grep(/^$self->{grant_object}$/, 'USER', 'TABLE', 'VIEW', 'MATERIALIZED VIEW', 'SEQUENCE', 'PROCEDURE', 'FUNCTION', 'PACKAGE BODY', 'TYPE', 'SYNONYM', 'DIRECTORY')); + } + + # Default boolean values + foreach my $k (keys %BOOLEAN_MAP) { + $self->{ora_boolean_values}{lc($k)} = $BOOLEAN_MAP{$k}; + } + # additional boolean values given from config file + foreach my $k (keys %{$self->{boolean_values}}) { + $self->{ora_boolean_values}{lc($k)} = $AConfig{BOOLEAN_VALUES}{$k}; + } + + # Set transaction isolation level + if ($self->{transaction} eq 'readonly') { + $self->{transaction} = 'SET TRANSACTION READ ONLY'; + } elsif ($self->{transaction} eq 'readwrite') { + $self->{transaction} = 'SET TRANSACTION READ WRITE'; + } elsif ($self->{transaction} eq 'committed') { + $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED'; + } elsif ($self->{transaction} eq 'serializable') { + $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE'; + } else { + if (grep(/^$self->{type}$/, 'COPY', 'INSERT')) { + $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE'; + } else { + $self->{transaction} = 'SET TRANSACTION ISOLATION LEVEL READ COMMITTED'; + } + } + $self->{function_check} = 1 if (not defined $self->{function_check} || $self->{function_check} eq ''); + $self->{qualify_function} = 1 if (!exists $self->{qualify_function}); + + # Set default function to use for uuid generation + $self->{uuid_function} ||= 'uuid_generate_v4'; + + # Set default cost unit value to 5 minutes + $self->{cost_unit_value} ||= 5; + + # Set default human days limit for type C migration level + $self->{human_days_limit} ||= 5; + + # Defined if column order must be optimized + $self->{reordering_columns} ||= 0; + + # Initialize suffix that may be added to the index name + $self->{indexes_suffix} ||= ''; + + # Disable synchronous commit for pg data load + $self->{synchronous_commit} ||= 0; + + # Disallow NOLOGGING / UNLOGGED table creation + $self->{disable_unlogged} ||= 0; + + # Default degree for Oracle parallelism + if ($self->{default_parallelism_degree} eq '') { + $self->{default_parallelism_degree} = 0; + } + + # Add header to output file + $self->{no_header} ||= 0; + + # Mark function as STABLE by default + if (not defined $self->{function_stable} || $self->{function_stable} ne '0') { + $self->{function_stable} = 1; + } + + # Initialize rewriting of index name + if (not defined $self->{indexes_renaming} || $self->{indexes_renaming} ne '0') { + $self->{indexes_renaming} = 1; + } + + # Enable autonomous transaction conversion. Default is enable it. + if (!exists $self->{autonomous_transaction} || $self->{autonomous_transaction} ne '0') { + $self->{autonomous_transaction} = 1; + } + + # Don't use *_pattern_ops with indexes by default + $self->{use_index_opclass} ||= 0; + + # Autodetect spatial type + $self->{autodetect_spatial_type} ||= 0; + + # Use btree_gin extenstion to create bitmap like index with pg >= 9.4 + $self->{bitmap_as_gin} = 1 if ($self->{bitmap_as_gin} ne '0'); + + # Create tables with OIDs or not, default to not create OIDs + $self->{with_oid} ||= 0; + + # Minimum of lines required in a table to use parallelism + $self->{parallel_min_rows} ||= 100000; + + # Should we replace zero date with something else than NULL + $self->{replace_zero_date} ||= ''; + if ($self->{replace_zero_date} && (uc($self->{replace_zero_date}) ne '-INFINITY') && ($self->{replace_zero_date} !~ /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$/)) { + die "FATAL: wrong format in REPLACE_ZERO_DATE value, should be YYYY-MM-DD HH:MM:SS or -INFINITY\n"; + } + + # Defined default value for to_number translation + $self->{to_number_conversion} ||= 'numeric'; + + # Set regexp to detect parts of statements that need to be considered as text + if ($AConfig{STRING_CONSTANT_REGEXP}) { + push(@{ $self->{string_constant_regexp} } , split(/;/, $AConfig{STRING_CONSTANT_REGEXP})); + } + if ($AConfig{ALTERNATIVE_QUOTING_REGEXP}) { + push(@{ $self->{alternative_quoting_regexp} } , split(/;/, $AConfig{ALTERNATIVE_QUOTING_REGEXP})); + } + + # Overwrite configuration with all given parameters + # and try to preserve backward compatibility + foreach my $k (keys %options) + { + if (($k eq 'allow') && $options{allow}) + { + $self->{limited} = (); + # Syntax: TABLE[regex1 regex2 ...];VIEW[regex1 regex2 ...];glob_regex1 glob_regex2 ... + my @allow_vlist = split(/\s*;\s*/, $options{allow}); + foreach my $a (@allow_vlist) + { + if ($a =~ /^([^\[]+)\[(.*)\]$/) { + push(@{$self->{limited}{"\U$1\E"}}, split(/[\s,]+/, $2) ); + } else { + push(@{$self->{limited}{ALL}}, split(/[\s,]+/, $a) ); + } + } + } + elsif (($k eq 'exclude') && $options{exclude}) + { + $self->{excluded} = (); + # Syntax: TABLE[regex1 regex2 ...];VIEW[regex1 regex2 ...];glob_regex1 glob_regex2 ... + my @exclude_vlist = split(/\s*;\s*/, $options{exclude}); + foreach my $a (@exclude_vlist) { + if ($a =~ /^([^\[]+)\[(.*)\]$/) { + push(@{$self->{excluded}{"\U$1\E"}}, split(/[\s,]+/, $2) ); + } else { + push(@{$self->{excluded}{ALL}}, split(/[\s,]+/, $a) ); + } + } + } + elsif (($k eq 'view_as_table') && $options{view_as_table}) + { + $self->{view_as_table} = (); + push(@{$self->{view_as_table}}, split(/[\s;,]+/, $options{view_as_table}) ); + } elsif (($k eq 'datasource') && $options{datasource}) { + $self->{oracle_dsn} = $options{datasource}; + } elsif (($k eq 'user') && $options{user}) { + $self->{oracle_user} = $options{user}; + } elsif (($k eq 'password') && $options{password}) { + $self->{oracle_pwd} = $options{password}; + } elsif (($k eq 'is_mysql') && $options{is_mysql}) { + $self->{is_mysql} = $options{is_mysql}; + } elsif ($options{$k} ne '') { + $self->{"\L$k\E"} = $options{$k}; + } + } + + # Global regex will be applied to the export type only + foreach my $i (@{$self->{limited}{ALL}}) + { + my $typ = $self->{type} || 'TABLE'; + $typ = 'TABLE' if ($self->{type} =~ /(SHOW_TABLE|SHOW_COLUMN|FDW|KETTLE|COPY|INSERT)/); + push(@{$self->{limited}{$typ}}, $i); + } + delete $self->{limited}{ALL}; + foreach my $i (@{$self->{excluded}{ALL}}) + { + my $typ = $self->{type} || 'TABLE'; + $typ = 'TABLE' if ($self->{type} =~ /(SHOW_TABLE|SHOW_COLUMN|FDW|KETTLE|COPY|INSERT)/); + push(@{$self->{excluded}{$typ}}, $i); + } + delete $self->{excluded}{ALL}; + + $self->{debug} = 1 if ($AConfig{'DEBUG'} == 1); + + # Set default XML data extract method + if (not defined $self->{xml_pretty} || ($self->{xml_pretty} != 0)) { + $self->{xml_pretty} = 1; + } + if (!$self->{fdw_server}) { + $self->{fdw_server} = 'orcl'; + } + + # Should we use \i or \ir in psql scripts + if ($AConfig{PSQL_RELATIVE_PATH}) { + $self->{psql_relative_path} = 'r'; + } else { + $self->{psql_relative_path} = ''; + } + + # Clean potential remaining temporary files + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + unlink($dirprefix . 'temp_pass2_file.dat'); + unlink($dirprefix . 'temp_cost_file.dat'); + + # Log file handle + $self->{fhlog} = undef; + if ($self->{logfile}) { + $self->{fhlog} = new IO::File; + $self->{fhlog}->open(">>$self->{logfile}") or $self->logit("FATAL: can't log to $self->{logfile}, $!\n", 0, 1); + } + + # Autoconvert SRID + if (not defined $self->{convert_srid} || ($self->{convert_srid} != 0)) { + $self->{convert_srid} = 1; + } + if (not defined $self->{default_srid}) { + $self->{default_srid} = 4326; + } + + # Force Ora2Pg to extract spatial object in binary format + $self->{geometry_extract_type} = uc($self->{geometry_extract_type}); + if (!$self->{geometry_extract_type} || !grep(/^$self->{geometry_extract_type}$/, 'WKT','WKB','INTERNAL')) { + $self->{geometry_extract_type} = 'INTERNAL'; + } + + # Default value for triming can be LEADING, TRAILING or BOTH + $self->{trim_type} = 'BOTH' if (!$self->{trim_type} || !grep(/^$self->{trim_type}/, 'BOTH', 'LEADING', 'TRAILING')); + # Default triming character is space + $self->{trim_char} = ' ' if ($self->{trim_char} eq ''); + + # Disable the use of orafce library by default + $self->{use_orafce} ||= 0; + + # Enable BLOB data export by default + if (not defined $self->{enable_blob_export}) { + $self->{enable_blob_export} = 1; + } + + # Table data export will be sorted by name by default + $self->{data_export_order} ||= 'name'; + + # Free some memory + %options = (); + %AConfig = (); + + # Enable create or replace by default + if ($self->{create_or_replace} || not defined $self->{create_or_replace}) { + $self->{create_or_replace} = ' OR REPLACE'; + } else { + $self->{create_or_replace} = ''; + } + + $self->{copy_freeze} = ' FREEZE' if ($self->{copy_freeze}); + # Prevent use of COPY FREEZE with some incompatible case + if ($self->{copy_freeze}) { + if ($self->{pg_dsn} && ($self->{jobs} > 1)) { + $self->logit("FATAL: You can not use COPY FREEZE with -j (JOBS) > 1 and direct import to PostgreSQL.\n", 0, 1); + } elsif ($self->{oracle_copies} > 1) { + $self->logit("FATAL: You can not use COPY FREEZE with -J (ORACLE_COPIES) > 1.\n", 0, 1); + } + } else { + $self->{copy_freeze} = ''; + } + + # Multiprocess init + $self->{jobs} ||= 1; + $self->{child_count} = 0; + # backward compatibility + if ($self->{thread_count}) { + $self->{jobs} = $self->{thread_count} || 1; + } + $self->{has_utf8_fct} = 1; + eval { utf8::valid("test utf8 function"); }; + if ($@) { + # Old perl install doesn't include these functions + $self->{has_utf8_fct} = 0; + } + + # Autodetexct if we are exporting a MySQL database + if ($self->{oracle_dsn} =~ /dbi:mysql/i) { + $self->{is_mysql} = 1; + } + + if ($self->{is_mysql}) { + # MySQL do not supports this syntax fallback to read committed + $self->{transaction} =~ s/(READ ONLY|READ WRITE)/ISOLATION LEVEL READ COMMITTED/; + } + + # Set Oracle, Perl and PostgreSQL encoding that will be used + $self->_init_environment(); + + # Multiple Oracle connection + $self->{oracle_copies} ||= 0; + $self->{ora_conn_count} = 0; + $self->{data_limit} ||= 10000; + $self->{blob_limit} ||= 0; + $self->{disable_partition} ||= 0; + $self->{parallel_tables} ||= 0; + $self->{no_lob_locator} = 1 if ($self->{no_lob_locator} ne '0'); + + # Transformation and output during data export + $self->{oracle_speed} ||= 0; + $self->{ora2pg_speed} ||= 0; + if (($self->{oracle_speed} || $self->{ora2pg_speed}) && !grep(/^$self->{type}$/, 'COPY', 'INSERT', 'DATA')) { + # No output is only available for data export. + die "FATAL: --oracle_speed or --ora2pg_speed can only be use with data export.\n"; + } + $self->{oracle_speed} = 1 if ($self->{ora2pg_speed}); + + # Shall we prefix function with a schema name to emulate a package? + $self->{package_as_schema} = 1 if (not exists $self->{package_as_schema} || ($self->{package_as_schema} eq '')); + $self->{package_functions} = (); + + # Set user defined data type translation + if ($self->{data_type}) { + $self->{data_type} =~ s/\\,/#NOSEP#/gs; + my @transl = split(/[,;]/, uc($self->{data_type})); + $self->{data_type} = (); + # Set default type conversion + %{$self->{data_type}} = %TYPE; + if ($self->{is_mysql}) { + %{$self->{data_type}} = %Ora2Pg::MySQL::MYSQL_TYPE; + } + # then set custom type conversion from the DATA_TYPE + # configuration directive + foreach my $t (@transl) { + my ($typ, $val) = split(/:/, $t); + $typ =~ s/^\s+//; + $typ =~ s/\s+$//; + $val =~ s/^\s+//; + $val =~ s/\s+$//; + $typ =~ s/#NOSEP#/,/g; + $val =~ s/#NOSEP#/,/g; + $self->{data_type}{$typ} = lc($val) if ($val); + } + } else { + # Set default type conversion + %{$self->{data_type}} = %TYPE; + if ($self->{is_mysql}) { + %{$self->{data_type}} = %Ora2Pg::MySQL::MYSQL_TYPE; + } + } + + # Set some default + $self->{global_where} ||= ''; + $self->{global_delete} ||= ''; + $self->{prefix} = 'DBA'; + if ($self->{user_grants}) { + $self->{prefix} = 'ALL'; + } + $self->{bzip2} ||= '/usr/bin/bzip2'; + $self->{default_numeric} ||= 'bigint'; + $self->{type_of_type} = (); + $self->{dump_as_html} ||= 0; + $self->{dump_as_csv} ||= 0; + $self->{dump_as_sheet} ||= 0; + $self->{top_max} ||= 10; + $self->{print_header} ||= 0; + $self->{use_default_null} = 1 if (!defined $self->{use_default_null}); + + $self->{estimate_cost} = 1 if ($self->{dump_as_sheet}); + $self->{count_rows} ||= 0; + + # Enforce preservation of primary and unique keys + # when USE_TABLESPACE is enabled + if ($self->{use_tablespace} && !$self->{keep_pkey_names}) { + print STDERR "WARNING: Enforcing KEEP_PKEY_NAMES to 1 as USE_TABLESPACE is enabled.\n"; + $self->{keep_pkey_names} = 1; + } + + # DATADIFF defaults + $self->{datadiff} ||= 0; + $self->{datadiff_del_suffix} ||= '_del'; + $self->{datadiff_ins_suffix} ||= '_ins'; + $self->{datadiff_upd_suffix} ||= '_upd'; + + # Internal date boundary. Date below will be added to 2000, others will used 1900 + $self->{internal_date_max} ||= 49; + + # Set the target PostgreSQL major version + if (!$self->{pg_version}) + { + print STDERR "WARNING: target PostgreSQL version must be set in PG_VERSION configuration directive. Using default: 11\n"; + $self->{pg_version} = 11; + } + + # Compatibility with PostgreSQL versions + if ($self->{pg_version} >= 9.0) { + $self->{pg_supports_when} = 1; + $self->{pg_supports_ifexists} = 'IF EXISTS'; + } + if ($self->{pg_version} >= 9.1) { + $self->{pg_supports_insteadof} = 1; + } + if ($self->{pg_version} >= 9.3) { + $self->{pg_supports_mview} = 1; + $self->{pg_supports_lateral} = 1; + } + if ($self->{pg_version} >= 9.4) { + $self->{pg_supports_checkoption} = 1; + } + if ($self->{pg_version} >= 9.5) { + $self->{pg_supports_named_operator} = 1; + } + if ($self->{pg_version} >= 10) { + $self->{pg_supports_partition} = 1; + $self->{pg_supports_identity} = 1; + } + if ($self->{pg_version} >= 11) { + $self->{pg_supports_procedure} = 1; + } + + # Other PostgreSQL fork compatibility + # Redshift + if ($self->{pg_supports_substr} eq '') { + $self->{pg_supports_substr} = 1; + } + + $self->{pg_background} ||= 0; + + # Backward compatibility with LongTrunkOk with typo + if ($self->{longtrunkok} && not defined $self->{longtruncok}) { + $self->{longtruncok} = $self->{longtrunkok}; + } + $self->{longtruncok} = 0 if (not defined $self->{longtruncok}); + # With lob locators LONGREADLEN must at least be 1MB + if (!$self->{longreadlen} || !$self->{no_lob_locator}) { + $self->{longreadlen} = (1023*1024); + } + + # Backward compatibility with PG_NUMERIC_TYPE alone + $self->{pg_integer_type} = 1 if (not defined $self->{pg_integer_type}); + # Backward compatibility with CASE_SENSITIVE + $self->{preserve_case} = $self->{case_sensitive} if (defined $self->{case_sensitive} && not defined $self->{preserve_case}); + $self->{schema} = uc($self->{schema}) if (!$self->{preserve_case} && ($self->{oracle_dsn} !~ /:mysql/i)); + # With MySQL override schema with the database name + if ($self->{oracle_dsn} =~ /:mysql:.*database=([^;]+)/i) { + if ($self->{schema} ne $1) { + $self->{schema} = $1; + #$self->logit("WARNING: setting SCHEMA to MySQL database name $1.\n", 0); + } + if (!$self->{schema}) { + $self->logit("FATAL: cannot find a valid mysql database in DSN, $self->{oracle_dsn}.\n", 0, 1); + } + } + + if (($self->{standard_conforming_strings} =~ /^off$/i) || ($self->{standard_conforming_strings} == 0)) { + $self->{standard_conforming_strings} = 0; + } else { + $self->{standard_conforming_strings} = 1; + } + if (!defined $self->{compile_schema} || $self->{compile_schema}) { + $self->{compile_schema} = 1; + } else { + $self->{compile_schema} = 0; + } + $self->{export_invalid} ||= 0; + $self->{use_reserved_words} ||= 0; + $self->{pkey_in_create} ||= 0; + $self->{security} = (); + # Should we add SET ON_ERROR_STOP to generated SQL files + $self->{stop_on_error} = 1 if (not defined $self->{stop_on_error}); + # Force foreign keys to be created initialy deferred if export type + # is TABLE or to set constraint deferred with data export types/ + $self->{defer_fkey} ||= 0; + + # Allow multiple or chained extraction export type + $self->{export_type} = (); + if ($self->{type}) { + @{$self->{export_type}} = split(/[\s,;]+/, $self->{type}); + # Assume backward comaptibility with DATA replacement by INSERT + map { s/^DATA$/INSERT/; } @{$self->{export_type}}; + } else { + @{$self->{export_type}} = ('TABLE'); + } + + # If you decide to autorewrite PLSQL code, this load the dedicated + # Perl module + $self->{plsql_pgsql} = 1 if ($self->{plsql_pgsql} eq ''); + $self->{plsql_pgsql} = 1 if ($self->{estimate_cost}); + if ($self->{plsql_pgsql}) { + use Ora2Pg::PLSQL; + } + + $self->{fhout} = undef; + $self->{compress} = ''; + $self->{pkgcost} = 0; + $self->{total_pkgcost} = 0; + + if ($^O =~ /MSWin32|dos/i) { + if ( ($self->{oracle_copies} > 1) || ($self->{jobs} > 1) || ($self->{parallel_tables} > 1) ) { + $self->logit("WARNING: multiprocess is not supported under that kind of OS.\n", 0); + $self->logit("If you need full speed at data export, please use Linux instead.\n", 0); + } + $self->{oracle_copies} = 0; + $self->{jobs} = 0; + $self->{parallel_tables} = 0; + } + if ($self->{parallel_tables} > 1) { + $self->{file_per_table} = 1; + } + if ($self->{jobs} > 1) { + $self->{file_per_function} = 1; + } + + if ($self->{debug}) { + $self->logit("Ora2Pg version: $VERSION\n"); + $self->logit("Export type: $self->{type}\n", 1); + $self->logit("Geometry export type: $self->{geometry_extract_type}\n", 1); + } + + # Replace ; or space by comma in the audit user list + $self->{audit_user} =~ s/[;\s]+/,/g; + + # Set the PostgreSQL connection information for data import or to + # defined the dblink connection to use in autonomous transaction + $self->set_pg_conn_details(); + + if (!$self->{input_file}) { + if ($self->{type} eq 'LOAD') { + $self->logit("FATAL: with LOAD you must provide an input file\n", 0, 1); + } + if (!$self->{oracle_dsn} || ($self->{oracle_dsn} =~ /;sid=SIDNAME/)) { + $self->logit("FATAL: you must set ORACLE_DSN in ora2pg.conf or use a DDL input file.\n", 0, 1); + } + # Connect the database + if ($self->{oracle_dsn} =~ /dbi:mysql/i) { + $self->{dbh} = $self->_mysql_connection(); + + $self->{is_mysql} = 1; + + # Get the Oracle version + $self->{db_version} = $self->_get_version(); + + } else { + $self->{dbh} = $self->_oracle_connection(); + + # Get the Oracle version + $self->{db_version} = $self->_get_version(); + + # Compile again all objects in the schema + if ($self->{compile_schema}) { + $self->_compile_schema(uc($self->{compile_schema})); + } + } + if (!grep(/^$self->{type}$/, 'COPY', 'INSERT', 'SEQUENCE', 'GRANT', 'TABLESPACE', 'QUERY', 'SYNONYM', 'FDW', 'KETTLE', 'DBLINK', 'DIRECTORY') && $self->{type} !~ /SHOW_/) { + if ($self->{plsql_pgsql} && !$self->{no_function_metadata}) { + my @done = (); + if ($#{ $self->{look_forward_function} } >= 0) { + foreach my $o (@{ $self->{look_forward_function} }) { + next if (grep(/^$o$/i, @done) || uc($o) eq uc($self->{schema})); + push(@done, $o); + if ($self->{type} eq 'VIEW') { + # Limit to package lookup with VIEW export type + $self->_get_package_function_list($o) if (!$self->{is_mysql}); + } else { + # Extract all package/function/procedure meta information + $self->_get_plsql_metadata($o); + } + } + } + if ($self->{type} eq 'VIEW') { + # Limit to package lookup with WIEW export type + $self->_get_package_function_list() if (!$self->{is_mysql}); + } else { + # Extract all package/function/procedure meta information + $self->_get_plsql_metadata(); + } + } + + $self->{security} = $self->_get_security_definer($self->{type}) if (grep(/^$self->{type}$/, 'TRIGGER', 'FUNCTION','PROCEDURE','PACKAGE')); + } + + } else { + + $self->{plsql_pgsql} = 1; + + if (grep(/^$self->{type}$/, 'TABLE', 'SEQUENCE', 'GRANT', 'TABLESPACE', 'VIEW', 'TRIGGER', 'QUERY', 'FUNCTION','PROCEDURE','PACKAGE','TYPE','SYNONYM', 'DIRECTORY', 'DBLINK','LOAD')) { + if ($self->{type} eq 'LOAD') { + if (!$self->{pg_dsn}) { + $self->logit("FATAL: You must set PG_DSN to connect to PostgreSQL to be able to dispatch load over multiple connections.\n", 0, 1); + } elsif ($self->{jobs} <= 1) { + $self->logit("FATAL: You must set set -j (JOBS) > 1 to be able to dispatch load over multiple connections.\n", 0, 1); + } + } + $self->export_schema(); + } else { + $self->logit("FATAL: bad export type using input file option\n", 0, 1); + } + return; + } + + # Register export structure modification + if ($self->{type} =~ /^(INSERT|COPY|TABLE)$/) { + for my $t (keys %{$self->{'modify_struct'}}) { + $self->modify_struct($t, @{$self->{'modify_struct'}{$t}}); + } + } + + # backup output filename in multiple export mode + $self->{output_origin} = ''; + if ($#{$self->{export_type}} > 0) { + $self->{output_origin} = $self->{output}; + } + + # Retreive all export types information + foreach my $t (@{$self->{export_type}}) + { + $self->{type} = $t; + + if (($self->{type} eq 'TABLE') || ($self->{type} eq 'FDW') || ($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY') || ($self->{type} eq 'KETTLE')) { + $self->{plsql_pgsql} = 1; + $self->_tables(); + # Partitionned table do not accept NOT VALID constraint + if ($self->{pg_supports_partition} && $self->{type} eq 'TABLE') { + # Get the list of partition + $self->{partitions} = $self->_get_partitions_list(); + } + } elsif ($self->{type} eq 'VIEW') { + $self->_views(); + } elsif ($self->{type} eq 'SYNONYM') { + $self->_synonyms(); + } elsif ($self->{type} eq 'GRANT') { + $self->_grants(); + } elsif ($self->{type} eq 'SEQUENCE') { + $self->_sequences(); + } elsif ($self->{type} eq 'TRIGGER') { + $self->_triggers(); + } elsif ($self->{type} eq 'FUNCTION') { + $self->_functions(); + } elsif ($self->{type} eq 'PROCEDURE') { + $self->_procedures(); + } elsif ($self->{type} eq 'PACKAGE') { + $self->_packages(); + } elsif ($self->{type} eq 'TYPE') { + $self->_types(); + } elsif ($self->{type} eq 'TABLESPACE') { + $self->_tablespaces(); + } elsif ($self->{type} eq 'PARTITION') { + $self->_partitions(); + } elsif ($self->{type} eq 'DBLINK') { + $self->_dblinks(); + } elsif ($self->{type} eq 'DIRECTORY') { + $self->_directories(); + } elsif ($self->{type} eq 'MVIEW') { + $self->_materialized_views(); + } elsif ($self->{type} eq 'QUERY') { + $self->_queries(); + } elsif ( ($self->{type} eq 'SHOW_REPORT') || ($self->{type} eq 'SHOW_VERSION') + || ($self->{type} eq 'SHOW_SCHEMA') || ($self->{type} eq 'SHOW_TABLE') + || ($self->{type} eq 'SHOW_COLUMN') || ($self->{type} eq 'SHOW_ENCODING')) + { + $self->_show_infos($self->{type}); + $self->{dbh}->disconnect() if ($self->{dbh}); + exit 0; + } elsif ($self->{type} eq 'TEST') { + $self->{dbhdest} = $self->_send_to_pgdb() if ($self->{pg_dsn} && !$self->{dbhdest}); + # Check if all tables have the same number of indexes, constraints, etc. + $self->_test_table(); + # Count each object at both sides + foreach my $o ('VIEW', 'MVIEW', 'SEQUENCE', 'TYPE', 'FDW') { + next if ($self->{is_mysql} && grep(/^$o$/, 'MVIEW','TYPE','FDW')); + $self->_count_object($o); + } + # count function/procedure/package function + $self->_test_function(); + # Count row in each table + if ($self->{count_rows}) { + $self->_table_row_count(); + } + $self->{dbh}->disconnect() if ($self->{dbh}); + exit 0; + } elsif ($self->{type} eq 'TEST_VIEW') { + $self->{dbhdest} = $self->_send_to_pgdb() if ($self->{pg_dsn} && !$self->{dbhdest}); + $self->_unitary_test_views(); + $self->{dbh}->disconnect() if ($self->{dbh}); + exit 0; + } else { + warn "type option must be (TABLE, VIEW, GRANT, SEQUENCE, TRIGGER, PACKAGE, FUNCTION, PROCEDURE, PARTITION, TYPE, INSERT, COPY, TABLESPACE, SHOW_REPORT, SHOW_VERSION, SHOW_SCHEMA, SHOW_TABLE, SHOW_COLUMN, SHOW_ENCODING, FDW, MVIEW, QUERY, KETTLE, DBLINK, SYNONYM, DIRECTORY, LOAD, TEST, TEST_VIEW), unknown $self->{type}\n"; + } + $self->replace_tables(%{$self->{'replace_tables'}}); + $self->replace_cols(%{$self->{'replace_cols'}}); + $self->set_where_clause($self->{'global_where'}, %{$self->{'where'}}); + $self->set_delete_clause($self->{'global_delete'}, %{$self->{'delete'}}); + } + + if ( ($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY') || ($self->{type} eq 'KETTLE') ) { + if ( ($self->{type} eq 'KETTLE') && !$self->{pg_dsn} ) { + $self->logit("FATAL: PostgreSQL connection datasource must be defined with KETTLE export.\n", 0, 1); + } elsif ($self->{type} ne 'KETTLE') { + if ($self->{defer_fkey} && $self->{pg_dsn}) { + $self->logit("FATAL: DEFER_FKEY can not be used with direct import to PostgreSQL, check use of DROP_FKEY instead.\n", 0, 1); + } + if ($self->{datadiff} && $self->{pg_dsn}) { + $self->logit("FATAL: DATADIFF can not be used with direct import to PostgreSQL because direct import may load data in several transactions.\n", 0, 1); + } + if ($self->{datadiff} && !$self->{pg_supports_lateral}) { + $self->logit("FATAL: DATADIFF requires LATERAL support (Pg version 9.3 and above; see config parameter PG_SUPPORTS_LATERAL)\n", 0, 1); + } + $self->{dbhdest} = $self->_send_to_pgdb() if ($self->{pg_dsn} && !$self->{dbhdest}); + } + } + + # Disconnect from the database + $self->{dbh}->disconnect() if ($self->{dbh}); + +} + + +sub _oracle_connection +{ + my ($self, $quiet) = @_; + + if (!defined $self->{oracle_pwd}) + { + eval("use Term::ReadKey;") unless $self->{oracle_user} eq '/'; + $self->{oracle_user} = $self->_ask_username('Oracle') unless (defined $self->{oracle_user}); + $self->{oracle_pwd} = $self->_ask_password('Oracle') unless ($self->{oracle_user} eq '/'); + } + my $ora_session_mode = ($self->{oracle_user} eq "/" || $self->{oracle_user} eq "sys") ? 2 : undef; + + $self->logit("ORACLE_HOME = $ENV{ORACLE_HOME}\n", 1); + $self->logit("NLS_LANG = $ENV{NLS_LANG}\n", 1); + $self->logit("NLS_NCHAR = $ENV{NLS_NCHAR}\n", 1); + $self->logit("Trying to connect to database: $self->{oracle_dsn}\n", 1) if (!$quiet); + + my $dbh = DBI->connect($self->{oracle_dsn}, $self->{oracle_user}, $self->{oracle_pwd}, + { + ora_envhp => 0, + LongReadLen=>$self->{longreadlen}, + LongTruncOk=>$self->{longtruncok}, + AutoInactiveDestroy => 1, + PrintError => 0, + ora_session_mode => $ora_session_mode, + ora_client_info => 'ora2pg ' || $VERSION + } + ); + + # Check for connection failure + if (!$dbh) { + $self->logit("FATAL: $DBI::err ... $DBI::errstr\n", 0, 1); + } + + # Get Oracle version, needed to set date/time format + my $sth = $dbh->prepare( "SELECT BANNER FROM v\$version" ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + $self->{db_version} = $row[0]; + last; + } + $sth->finish(); + chomp($self->{db_version}); + $self->{db_version} =~ s/ \- .*//; + + # Check if the connection user has the DBA privilege + $sth = $dbh->prepare( "SELECT 1 FROM DBA_ROLE_PRIVS" ); + if (!$sth) { + my $ret = $dbh->err; + if ($ret == 942 && $self->{prefix} eq 'DBA') { + $self->logit("HINT: you should activate USER_GRANTS for a connection without DBA privilege. Continuing with USER privilege.\n"); + # No DBA privilege, set use of ALL_* tables instead of DBA_* tables + $self->{prefix} = 'ALL'; + $self->{user_grants} = 1; + } + } else { + $sth->finish(); + } + + # Fix a problem when exporting type LONG and LOB + $dbh->{'LongReadLen'} = $self->{longreadlen}; + $dbh->{'LongTruncOk'} = $self->{longtruncok}; + # Embedded object (user defined type) must be returned as an + # array rather than an instance. This is normally the default. + $dbh->{'ora_objects'} = 0; + + # Force datetime format + $self->_datetime_format($dbh); + # Force numeric format + $self->_numeric_format($dbh); + + # Use consistent reads for concurrent dumping... + $dbh->begin_work || $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + if ($self->{debug} && !$quiet) { + $self->logit("Isolation level: $self->{transaction}\n", 1); + } + $sth = $dbh->prepare($self->{transaction}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->finish; + + # Force execution of initial command + $self->_ora_initial_command($dbh); + + return $dbh; +} + +sub _mysql_connection +{ + my ($self, $quiet) = @_; + + use Ora2Pg::MySQL; + + $self->logit("Trying to connect to database: $self->{oracle_dsn}\n", 1) if (!$quiet); + + if (!defined $self->{oracle_pwd}) + { + eval("use Term::ReadKey;"); + $self->{oracle_user} = $self->_ask_username('MySQL') unless (defined $self->{oracle_user}); + $self->{oracle_pwd} = $self->_ask_password('MySQL'); + } + + my $dbh = DBI->connect("$self->{oracle_dsn}", $self->{oracle_user}, $self->{oracle_pwd}, { + 'RaiseError' => 1, + AutoInactiveDestroy => 1, + mysql_enable_utf8 => 1, + mysql_conn_attrs => { program_name => 'ora2pg ' || $VERSION } + } + ); + + # Check for connection failure + if (!$dbh) { + $self->logit("FATAL: $DBI::err ... $DBI::errstr\n", 0, 1); + } + + # Use consistent reads for concurrent dumping... + #$dbh->do('START TRANSACTION WITH CONSISTENT SNAPSHOT;') || $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + if ($self->{debug} && !$quiet) { + $self->logit("Isolation level: $self->{transaction}\n", 1); + } + my $sth = $dbh->prepare($self->{transaction}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->finish; + + # Get SQL_MODE from the MySQL database + $sth = $dbh->prepare('SELECT @@sql_mode') or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + $self->{mysql_mode} = $row->[0]; + } + $sth->finish; + + if ($self->{nls_lang}) { + if ($self->{debug} && !$quiet) { + $self->logit("Set default encoding to '$self->{nls_lang}' and collate to '$self->{nls_nchar}'\n", 1); + } + my $collate = ''; + $collate = " COLLATE '$self->{nls_nchar}'" if ($self->{nls_nchar}); + $sth = $dbh->prepare("SET NAMES '$self->{nls_lang}'$collate") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->finish; + } + # Force execution of initial command + $self->_ora_initial_command($dbh); + + if ($self->{mysql_mode} =~ /PIPES_AS_CONCAT/) { + $self->{mysql_pipes_as_concat} = 1; + } + + # Instruct Ora2Pg that the database engine is mysql + $self->{is_mysql} = 1; + + return $dbh; +} + +# use to set encoding +sub _init_environment +{ + my ($self) = @_; + + # Set default Oracle client encoding + if (!$self->{nls_lang}) { + if (!$self->{is_mysql}) { + $self->{nls_lang} = 'AMERICAN_AMERICA.AL32UTF8'; + } else { + $self->{nls_lang} = 'utf8'; + } + } + if (!$self->{nls_nchar}) { + if (!$self->{is_mysql}) { + $self->{nls_nchar} = 'AL32UTF8'; + } else { + $self->{nls_nchar} = 'utf8_general_ci'; + } + } + $ENV{NLS_LANG} = $self->{nls_lang}; + $ENV{NLS_NCHAR} = $self->{nls_nchar}; + + # Force Perl to use utf8 I/O encoding by default or the + # encoding given in the BINMODE configuration directive. + # See http://perldoc.perl.org/5.14.2/open.html for values + # that can be used. Default is :utf8 + $self->set_binmode(); + + # Set default PostgreSQL client encoding to UTF8 + if (!$self->{client_encoding} || ($self->{nls_lang} =~ /UTF8/) ) { + $self->{client_encoding} = 'UTF8'; + } + +} + +sub set_binmode +{ + my $self = shift; + + my ($package, $filename, $line) = caller; + + if ( !$self->{input_file} && (!$self->{'binmode'} || $self->{nls_lang} =~ /UTF8/i) ) { + use open ':utf8'; + } elsif ($self->{'binmode'} =~ /^:/) { + eval "use open '$self->{'binmode'}';" or die "FATAL: can't use open layer $self->{'binmode'}\n"; + } elsif ($self->{'binmode'}) { + eval "use open 'encoding($self->{'binmode'})';" or die "FATAL: can't use open layer :encoding($self->{'binmode'})\n"; + } + # Set default PostgreSQL client encoding to UTF8 + if (!$self->{client_encoding} || ($self->{nls_lang} =~ /UTF8/ && !$self->{input_file}) ) { + $self->{client_encoding} = 'UTF8'; + } + + if ($#_ == 0) { + my $enc = $self->{'binmode'} || 'utf8'; + $enc =~ s/^://; + binmode($_[0], ":encoding($enc)"); + } + +} + +sub _is_utf8_file +{ + + my $file = shift(); + + my $utf8 = 0; + if (open(my $f, '<', $file)) { + local $/; + my $data = <$f>; + close($f); + if (utf8::decode($data)) { + $utf8 = 1 + } + } + + return $utf8; +} + +# We provide a DESTROY method so that the autoloader doesn't +# bother trying to find it. We also close the DB connexion +sub DESTROY +{ + my $self = shift; + + #$self->{dbh}->disconnect() if ($self->{dbh}); + +} + + +sub set_pg_conn_details +{ + my $self = shift; + + # Init connection details with configuration options + $self->{pg_dsn} ||= ''; + + $self->{pg_dsn} =~ /dbname=([^;]*)/; + $self->{dbname} = $1 || 'testdb'; + $self->{pg_dsn} =~ /host=([^;]*)/; + $self->{dbhost} = $1 || 'localhost'; + $self->{pg_dsn} =~ /port=([^;]*)/; + $self->{dbport} = $1 || 5432; + $self->{dbuser} = $self->{pg_user} || 'pguser'; + $self->{dbpwd} = $self->{pg_pwd} || 'pgpwd'; + + if (!$self->{dblink_conn}) { + #$self->{dblink_conn} = "port=$self->{dbport} dbname=$self->{dbname} host=$self->{dbhost} user=$self->{dbuser} password=$self->{dbpwd}"; + # Use a more generic connection string, the password must be + # set in .pgpass. Default is to use unix socket to connect. + $self->{dblink_conn} = "format('port=%s dbname=%s user=%s', current_setting('port'), current_database(), current_user)"; + } +} + + +=head2 _send_to_pgdb + +Open a DB handle to a PostgreSQL database + +=cut + +sub _send_to_pgdb +{ + my ($self) = @_; + + eval("use DBD::Pg qw(:pg_types);"); + + return if ($self->{oracle_speed}); + + if (!defined $self->{pg_pwd}) + { + eval("use Term::ReadKey;"); + $self->{pg_user} = $self->_ask_username('PostgreSQL') unless (defined($self->{pg_user})); + $self->{pg_pwd} = $self->_ask_password('PostgreSQL'); + } + + $ENV{PGAPPNAME} = 'ora2pg ' || $VERSION; + + # Connect the destination database + my $dbhdest = DBI->connect($self->{pg_dsn}, $self->{pg_user}, $self->{pg_pwd}, {AutoInactiveDestroy => 1}); + + # Check for connection failure + if (!$dbhdest) { + $self->logit("FATAL: $DBI::err ... $DBI::errstr\n", 0, 1); + } + + # Force execution of initial command + $self->_pg_initial_command($dbhdest); + + return $dbhdest; +} + +=head2 _grants + +This function is used to retrieve all privilege information. + +It extracts all Oracle's ROLES to convert them to Postgres groups (or roles) +and searches all users associated to these roles. + +=cut + +sub _grants +{ + my ($self) = @_; + + $self->logit("Retrieving users/roles/grants information...\n", 1); + ($self->{grants}, $self->{roles}) = $self->_get_privilege(); +} + + +=head2 _sequences + +This function is used to retrieve all sequences information. + +=cut + +sub _sequences +{ + my ($self) = @_; + + $self->logit("Retrieving sequences information...\n", 1); + $self->{sequences} = $self->_get_sequences(); + +} + + +=head2 _triggers + +This function is used to retrieve all triggers information. + +=cut + +sub _triggers +{ + my ($self) = @_; + + $self->logit("Retrieving triggers information...\n", 1); + $self->{triggers} = $self->_get_triggers(); +} + + +=head2 _functions + +This function is used to retrieve all functions information. + +=cut + +sub _functions +{ + my $self = shift; + + $self->logit("Retrieving functions information...\n", 1); + $self->{functions} = $self->_get_functions(); + +} + +=head2 _procedures + +This function is used to retrieve all procedures information. + +=cut + +sub _procedures +{ + my $self = shift; + + $self->logit("Retrieving procedures information...\n", 1); + + $self->{procedures} = $self->_get_procedures(); + +} + + +=head2 _packages + +This function is used to retrieve all packages information. + +=cut + +sub _packages +{ + my ($self) = @_; + + $self->logit("Retrieving packages information...\n", 1); + $self->{packages} = $self->_get_packages(); + +} + + +=head2 _types + +This function is used to retrieve all custom types information. + +=cut + +sub _types +{ + my ($self) = @_; + + $self->logit("Retrieving user defined types information...\n", 1); + $self->{types} = $self->_get_types(); + +} + +=head2 _tables + +This function is used to retrieve all table information. + +Sets the main hash of the database structure $self->{tables}. +Keys are the names of all tables retrieved from the current +database. Each table information is composed of an array associated +to the table_info key as array reference. In other way: + + $self->{tables}{$class_name}{table_info} = [(OWNER,TYPE,COMMENT,NUMROW)]; + +DBI TYPE can be TABLE, VIEW, SYSTEM TABLE, GLOBAL TEMPORARY, LOCAL TEMPORARY, +ALIAS, SYNONYM or a data source specific type identifier. This only extracts +the TABLE type. + +It also gets the following information in the DBI object to affect the +main hash of the database structure : + + $self->{tables}{$class_name}{field_name} = $sth->{NAME}; + $self->{tables}{$class_name}{field_type} = $sth->{TYPE}; + +It also calls these other private subroutines to affect the main hash +of the database structure : + + @{$self->{tables}{$class_name}{column_info}} = $self->_column_info($class_name, $owner, 'TABLE'); + %{$self->{tables}{$class_name}{unique_key}} = $self->_unique_key($class_name, $owner); + @{$self->{tables}{$class_name}{foreign_key}} = $self->_foreign_key($class_name, $owner); + %{$self->{tables}{$class_name}{check_constraint}} = $self->_check_constraint($class_name, $owner); + +=cut + +sub sort_view_by_iter +{ + + if (exists $ordered_views{$a}{iter} || exists $ordered_views{$b}{iter}) { + return $ordered_views{$a}{iter} <=> $ordered_views{$b}{iter}; + } else { + return $a cmp $b; + } +} + +sub _tables +{ + my ($self, $nodetail) = @_; + + # Get all tables information specified by the DBI method table_info + $self->logit("Retrieving table information...\n", 1); + + # Retrieve tables informations + my %tables_infos = $self->_table_info(); + + # Retrieve column identity information + if ($self->{type} ne 'FDW') + { + %{ $self->{identity_info} } = $self->_get_identities(); + } + + if (scalar keys %tables_infos > 0) + { + if ( grep(/^$self->{type}$/, 'TABLE','SHOW_REPORT','COPY','INSERT') + && !$self->{skip_indices} && !$self->{skip_indexes}) + { + $self->logit("Retrieving index information...\n", 1); + my $autogen = 0; + $autogen = 1 if (grep(/^$self->{type}$/, 'COPY','INSERT')); + my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}, $autogen); + foreach my $tb (keys %{$indexes}) + { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{indexes}} = %{$indexes->{$tb}}; + } + foreach my $tb (keys %{$idx_type}) { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{idx_type}} = %{$idx_type->{$tb}}; + } + foreach my $tb (keys %{$idx_tbsp}) { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{idx_tbsp}} = %{$idx_tbsp->{$tb}}; + } + foreach my $tb (keys %{$uniqueness}) { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{uniqueness}} = %{$uniqueness->{$tb}}; + } + } + + # Get detailed informations on each tables + if (!$nodetail) + { + $self->logit("Retrieving columns information...\n", 1); + # Retrieve all column's details + my %columns_infos = $self->_column_info('',$self->{schema}, 'TABLE'); + foreach my $tb (keys %columns_infos) + { + next if (!exists $tables_infos{$tb}); + foreach my $c (keys %{$columns_infos{$tb}}) { + push(@{$self->{tables}{$tb}{column_info}{$c}}, @{$columns_infos{$tb}{$c}}); + } + } + %columns_infos = (); + + # Retrieve comment of each columns and FK information if not foreign table export + if ($self->{type} ne 'FDW') + { + if ($self->{type} eq 'TABLE') + { + $self->logit("Retrieving comments information...\n", 1); + my %columns_comments = $self->_column_comments(); + foreach my $tb (keys %columns_comments) + { + next if (!exists $tables_infos{$tb}); + foreach my $c (keys %{$columns_comments{$tb}}) { + $self->{tables}{$tb}{column_comments}{$c} = $columns_comments{$tb}{$c}; + } + } + } + + # Extract foreign keys informations + if (!$self->{skip_fkeys}) + { + $self->logit("Retrieving foreign keys information...\n", 1); + my ($foreign_link, $foreign_key) = $self->_foreign_key('',$self->{schema}); + foreach my $tb (keys %{$foreign_link}) { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{foreign_link}} = %{$foreign_link->{$tb}}; + } + foreach my $tb (keys %{$foreign_key}) { + next if (!exists $tables_infos{$tb}); + push(@{$self->{tables}{$tb}{foreign_key}}, @{$foreign_key->{$tb}}); + } + } + } + } + + # Retrieve unique keys and check constraint information if not FDW export + if ($self->{type} ne 'FDW') + { + $self->logit("Retrieving unique keys information...\n", 1); + my %unique_keys = $self->_unique_key('',$self->{schema}); + foreach my $tb (keys %unique_keys) + { + next if (!exists $tables_infos{$tb}); + foreach my $c (keys %{$unique_keys{$tb}}) { + $self->{tables}{$tb}{unique_key}{$c} = $unique_keys{$tb}{$c}; + } + } + %unique_keys = (); + + if (!$self->{skip_checks} && !$self->{is_mysql}) + { + $self->logit("Retrieving check constraints information...\n", 1); + my %check_constraints = $self->_check_constraint('',$self->{schema}); + foreach my $tb (keys %check_constraints) { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{check_constraint}} = ( %{$check_constraints{$tb}}); + } + } + + } + } + + my @done = (); + my $id = 0; + # Set the table information for each class found + my $i = 1; + my $num_total_table = scalar keys %tables_infos; + my $count_table = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_table); + foreach my $t (sort keys %tables_infos) + { + if (!$self->{quiet} && !$self->{debug} && ($count_table % $PGBAR_REFRESH) == 0) + { + print STDERR $self->progress_bar($i, $num_total_table, 25, '=', 'tables', "scanning table $t" ), "\r"; + } + $count_table++; + + if (grep(/^$t$/, @done)) { + $self->logit("Duplicate entry found: $t\n", 1); + } else { + push(@done, $t); + } + $self->logit("[$i] Scanning table $t ($tables_infos{$t}{num_rows} rows)...\n", 1); + + # Check of uniqueness of the table + if (exists $self->{tables}{$t}{field_name}) { + $self->logit("Warning duplicate table $t, maybe a SYNONYM ? Skipped.\n", 1); + next; + } + # Try to respect order specified in the TABLES limited extraction array + if ($#{$self->{limited}{TABLE}} > 0) + { + $self->{tables}{$t}{internal_id} = 0; + for (my $j = 0; $j <= $#{$self->{limited}{TABLE}}; $j++) + { + if (uc($self->{limited}{TABLE}->[$j]) eq uc($t)) + { + $self->{tables}{$t}{internal_id} = $j; + last; + } + } + } + + # usually TYPE,COMMENT,NUMROW,... + $self->{tables}{$t}{table_info}{type} = $tables_infos{$t}{type}; + $self->{tables}{$t}{table_info}{comment} = $tables_infos{$t}{comment}; + $self->{tables}{$t}{table_info}{num_rows} = $tables_infos{$t}{num_rows}; + $self->{tables}{$t}{table_info}{owner} = $tables_infos{$t}{owner}; + $self->{tables}{$t}{table_info}{tablespace} = $tables_infos{$t}{tablespace}; + $self->{tables}{$t}{table_info}{nested} = $tables_infos{$t}{nested}; + $self->{tables}{$t}{table_info}{size} = $tables_infos{$t}{size}; + $self->{tables}{$t}{table_info}{auto_increment} = $tables_infos{$t}{auto_increment}; + $self->{tables}{$t}{table_info}{connection} = $tables_infos{$t}{connection}; + $self->{tables}{$t}{table_info}{nologging} = $tables_infos{$t}{nologging}; + $self->{tables}{$t}{table_info}{partitioned} = $tables_infos{$t}{partitioned}; + if (exists $tables_infos{$t}{fillfactor}) { + $self->{tables}{$t}{table_info}{fillfactor} = $tables_infos{$t}{fillfactor}; + } + + # Set the fields information + if ($self->{type} ne 'SHOW_REPORT') + { + my $tmp_tbname = $t; + if (!$self->{is_mysql}) + { + if ( $t !~ /\./ ) { + $tmp_tbname = "\"$tables_infos{$t}{owner}\".\"$t\""; + } else { + # in case we already have the schema name, add doublequote + $tmp_tbname =~ s/\./"."/; + $tmp_tbname = "\"$tmp_tbname\""; + } + } + my $query = "SELECT * FROM $tmp_tbname WHERE 1=0"; + if ($tables_infos{$t}{nested} eq 'YES') { + $query = "SELECT /*+ nested_table_get_refs */ * FROM $tmp_tbname WHERE 1=0"; + } + my $sth = $self->{dbh}->prepare($query); + if (!defined($sth)) { + warn "Can't prepare statement: $DBI::errstr"; + next; + } + $sth->execute; + if ($sth->err) { + warn "Can't execute statement: $DBI::errstr"; + next; + } + $self->{tables}{$t}{type} = 'table'; + $self->{tables}{$t}{field_name} = $sth->{NAME}; + $self->{tables}{$t}{field_type} = $sth->{TYPE}; + } + $i++; + } + + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_table, 25, '=', 'tables', 'end of scanning.'), "\n"; + } + + # Try to search requested TABLE names in the VIEW names if not found in + # real TABLE names + if ($#{$self->{view_as_table}} >= 0) + { + my %view_infos = $self->_get_views(); + # Retrieve comment of each columns + my %columns_comments = $self->_column_comments(); + foreach my $view (keys %columns_comments) { + next if (!exists $view_infos{$view}); + next if (!grep($view =~ /^$_$/i, @{$self->{view_as_table}})); + foreach my $c (keys %{$columns_comments{$view}}) { + $self->{tables}{$view}{column_comments}{$c} = $columns_comments{$view}{$c}; + } + } + foreach my $view (sort keys %view_infos) { + # Set the table information for each class found + # Jump to desired extraction + next if (!grep($view =~ /^$_$/i, @{$self->{view_as_table}})); + $self->logit("Scanning view $view to export as table...\n", 0); + + $self->{tables}{$view}{type} = 'view'; + $self->{tables}{$view}{text} = $view_infos{$view}{text}; + $self->{tables}{$view}{owner} = $view_infos{$view}{owner}; + $self->{tables}{$view}{iter} = $view_infos{$view}{iter} if (exists $view_infos{$view}{iter}); + $self->{tables}{$view}{alias}= $view_infos{$view}{alias}; + $self->{tables}{$view}{comment} = $view_infos{$view}{comment}; + my $realview = $view; + $realview =~ s/"//g; + if (!$self->{is_mysql}) { + if ($realview !~ /\./) { + $realview = "\"$self->{tables}{$view}{owner}\".\"$realview\""; + } else { + $realview =~ s/\./"."/; + $realview = "\"$realview\""; + } + + } + # Set the fields information + my $sth = $self->{dbh}->prepare("SELECT * FROM $realview WHERE 1=0"); + if (!defined($sth)) { + warn "Can't prepare statement: $DBI::errstr"; + next; + } + $sth->execute; + if ($sth->err) { + warn "Can't execute statement: $DBI::errstr"; + next; + } + $self->{tables}{$view}{field_name} = $sth->{NAME}; + $self->{tables}{$view}{field_type} = $sth->{TYPE}; + my %columns_infos = $self->_column_info($view, $self->{schema}, 'VIEW'); + foreach my $tb (keys %columns_infos) { + next if ($tb ne $view); + foreach my $c (keys %{$columns_infos{$tb}}) { + push(@{$self->{tables}{$view}{column_info}{$c}}, @{$columns_infos{$tb}{$c}}); + } + } + } + } + + # Look at external tables + if (!$self->{is_mysql} && ($self->{db_version} !~ /Release 8/)) { + %{$self->{external_table}} = $self->_get_external_tables(); + } + + if ($self->{type} eq 'TABLE') + { + $self->logit("Retrieving table partitioning information...\n", 0); + %{ $self->{partitions_list} } = $self->_get_partitioned_table(); + } +} + +sub _get_plsql_code +{ + my $str = shift(); + + my $ct = ''; + my @parts = split(/\b(BEGIN|DECLARE|END\s*(?!IF|LOOP|CASE|INTO|FROM|,|\))[^;\s]*\s*;)/i, $str); + my $code = ''; + my $other = ''; + my $i = 0; + for (; $i <= $#parts; $i++) + { + $ct++ if ($parts[$i] =~ /\bBEGIN\b/i); + $ct-- if ($parts[$i] =~ /\bEND\s*(?!IF|LOOP|CASE|INTO|FROM|,|\))[^;\s]*\s*;/i); + if ( ($ct ne '') && ($ct == 0) ) { + $code .= $parts[$i]; + last; + } + $code .= $parts[$i]; + } + $i++; + for (; $i <= $#parts; $i++) { + $other .= $parts[$i]; + } + + return ($code, $other); +} + +sub _parse_constraint +{ + my ($self, $tb_name, $cur_col_name, $c) = @_; + + if ($c =~ /^([^\s]+)\s+(UNIQUE|PRIMARY KEY)\s*\(([^\)]+)\)/is) { + my $tp = 'U'; + $tp = 'P' if ($2 eq 'PRIMARY KEY'); + $self->{tables}{$tb_name}{unique_key}{$1} = { ( + type => $tp, 'generated' => 0, 'index_name' => $1, + columns => () + ) }; + push(@{$self->{tables}{$tb_name}{unique_key}{$1}{columns}}, split(/\s*,\s*/, $3)); + } elsif ($c =~ /^([^\s]+)\s+CHECK\s*\((.*)\)/is) { + my %tmp = ($1 => $2); + $self->{tables}{$tb_name}{check_constraint}{constraint}{$1}{condition} = $2; + if ($c =~ /NOVALIDATE/is) { + $self->{tables}{$tb_name}{check_constraint}{constraint}{$1}{validate} = 'NOT VALIDATED'; + } + } elsif ($c =~ /^([^\s]+)\s+FOREIGN KEY (\([^\)]+\))?\s*REFERENCES ([^\(\s]+)\s*\(([^\)]+)\)/is) { + my $c_name = $1; + if ($2) { + $cur_col_name = $2; + } + my $f_tb_name = $3; + my @col_list = split(/,/, $4); + $c_name =~ s/"//g; + $f_tb_name =~ s/"//g; + $cur_col_name =~ s/[\("\)]//g; + map { s/"//g; } @col_list; + if (!$self->{export_schema}) { + $f_tb_name =~ s/^[^\.]+\.//; + map { s/^[^\.]+\.//; } @col_list; + } + push(@{$self->{tables}{$tb_name}{foreign_link}{"\U$c_name\E"}{local}}, $cur_col_name); + push(@{$self->{tables}{$tb_name}{foreign_link}{"\U$c_name\E"}{remote}{$f_tb_name}}, @col_list); + my $deferrable = ''; + $deferrable = 'DEFERRABLE' if ($c =~ /DEFERRABLE/); + my $deferred = ''; + $deferred = 'DEFERRED' if ($c =~ /INITIALLY DEFERRED/); + my $novalidate = ''; + $novalidate = 'NOT VALIDATED' if ($c =~ /NOVALIDATE/); + # CONSTRAINT_NAME,R_CONSTRAINT_NAME,SEARCH_CONDITION,DELETE_RULE,$deferrable,DEFERRED,R_OWNER,TABLE_NAME,OWNER,UPDATE_RULE,VALIDATED + push(@{$self->{tables}{$tb_name}{foreign_key}}, [ ($c_name,'','','',$deferrable,$deferred,'',$tb_name,'','',$novalidate) ]); + } +} + +sub _remove_text_constant_part +{ + my ($self, $str) = @_; + + for (my $i = 0; $i <= $#{$self->{alternative_quoting_regexp}}; $i++) { + while ($$str =~ s/$self->{alternative_quoting_regexp}[$i]/\?TEXTVALUE$self->{text_values_pos}\?/s) { + $self->{text_values}{$self->{text_values_pos}} = '$$' . $1 . '$$'; + $self->{text_values_pos}++; + } + } + + $$str =~ s/\\'/ORA2PG_ESCAPE1_QUOTE'/gs; + while ($$str =~ s/''/ORA2PG_ESCAPE2_QUOTE/gs) {} + + while ($$str =~ s/('[^']+')/\?TEXTVALUE$self->{text_values_pos}\?/s) { + $self->{text_values}{$self->{text_values_pos}} = $1; + $self->{text_values_pos}++; + } + + for (my $i = 0; $i <= $#{$self->{string_constant_regexp}}; $i++) { + while ($$str =~ s/($self->{string_constant_regexp}[$i])/\?TEXTVALUE$self->{text_values_pos}\?/s) { + $self->{text_values}{$self->{text_values_pos}} = $1; + $self->{text_values_pos}++; + } + } +} + +sub _restore_text_constant_part +{ + my ($self, $str) = @_; + + $$str =~ s/\?TEXTVALUE(\d+)\?/$self->{text_values}{$1}/gs; + $$str =~ s/ORA2PG_ESCAPE2_QUOTE/''/gs; + $$str =~ s/ORA2PG_ESCAPE1_QUOTE'/\\'/gs; + + if ($self->{type} eq 'TRIGGER') { + $$str =~ s/(\s+)(NEW|OLD)\.'([^']+)'/$1$2\.$3/igs; + } +} + +sub _get_dml_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->read_input_file($self->{input_file}); + + $content =~ s/CREATE\s+OR\s+REPLACE/CREATE/gs; + $content =~ s/CREATE\s+EDITIONABLE/CREATE/gs; + $content =~ s/CREATE\s+NONEDITIONABLE/CREATE/gs; + + if ($self->{is_mysql}) + { + $content =~ s/CREATE\s+ALGORITHM=[^\s]+/CREATE/gs; + $content =~ s/CREATE\s+DEFINER=[^\s]+/CREATE/gs; + $content =~ s/SQL SECURITY DEFINER VIEW/VIEW/gs; + } + + return $content; +} + +sub read_schema_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Clear content from comment and text constant for better parsing + $self->_remove_comments(\$content, 1); + $content =~ s/\%ORA2PG_COMMENT\d+\%//gs; + my $tid = 0; + + my @statements = split(/\s*;\s*/, $content); + + foreach $content (@statements) + { + $content .= ';'; + + # Remove some unwanted and unused keywords from the statements + $content =~ s/\s+(PARALLEL|COMPRESS)\b//igs; + + if ($content =~ s/TRUNCATE TABLE\s+([^\s;]+)([^;]*);//is) + { + my $tb_name = $1; + $tb_name =~ s/"//gs; + if (!exists $self->{tables}{$tb_name}{table_info}{type}) + { + $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; + $self->{tables}{$tb_name}{table_info}{num_rows} = 0; + $tid++; + $self->{tables}{$tb_name}{internal_id} = $tid; + } + $self->{tables}{$tb_name}{truncate_table} = 1; + } + elsif ($content =~ s/CREATE\s+(GLOBAL|PRIVATE)?\s*(TEMPORARY)?\s*TABLE[\s]+([^\s]+)\s+AS\s+([^;]+);//is) + { + my $tb_name = $3; + $tb_name =~ s/"//gs; + my $tb_def = $4; + $tb_def =~ s/\s+/ /gs; + $self->{tables}{$tb_name}{table_info}{type} = 'TEMPORARY ' if ($2); + $self->{tables}{$tb_name}{table_info}{type} .= 'TABLE'; + $self->{tables}{$tb_name}{table_info}{num_rows} = 0; + $tid++; + $self->{tables}{$tb_name}{internal_id} = $tid; + $self->{tables}{$tb_name}{table_as} = $tb_def; + } + elsif ($content =~ s/CREATE\s+(GLOBAL|PRIVATE)?\s*(TEMPORARY)?\s*TABLE[\s]+([^\s\(]+)\s*([^;]+);//is) + { + my $tb_name = $3; + my $tb_def = $4; + my $tb_param = ''; + $tb_name =~ s/"//gs; + $self->{tables}{$tb_name}{table_info}{type} = 'TEMPORARY ' if ($2); + $self->{tables}{$tb_name}{table_info}{type} .= 'TABLE'; + $self->{tables}{$tb_name}{table_info}{num_rows} = 0; + $tid++; + $self->{tables}{$tb_name}{internal_id} = $tid; + # For private temporary table extract the ON COMMIT information (18c) + if ($tb_def =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION//is) + { + $self->{tables}{$tb_name}{table_info}{on_commit} = 'ON COMMIT PRESERVE ROWS'; + } + elsif ($tb_def =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION//is) + { + $self->{tables}{$tb_name}{table_info}{on_commit} = 'ON COMMIT DROP'; + } + elsif ($self->{tables}{$tb_name}{table_info}{type} eq 'TEMPORARY ') + { + # Default for PRIVATE TEMPORARY TABLE + $self->{tables}{$tb_name}{table_info}{on_commit} = 'ON COMMIT DROP'; + } + # Get table embedded comment + if ($tb_def =~ s/COMMENT=["']([^"']+)["']//is) + { + $self->{tables}{$tb_name}{table_info}{comment} = $1; + } + $tb_def =~ s/^\(//; + my %fct_placeholder = (); + my $i = 0; + while ($tb_def =~ s/(\([^\(\)]*\))/\%\%FCT$i\%\%/is) + { + $fct_placeholder{$i} = $1; + $i++; + }; + ($tb_def, $tb_param) = split(/\s*\)\s*/, $tb_def); + my @column_defs = split(/\s*,\s*/, $tb_def); + map { s/^\s+//; s/\s+$//; } @column_defs; + my $pos = 0; + my $cur_c_name = ''; + foreach my $c (@column_defs) + { + next if (!$c); + + # Replace temporary substitution + while ($c =~ s/\%\%FCT(\d+)\%\%/$fct_placeholder{$1}/is) { + delete $fct_placeholder{$1}; + } + # Mysql unique key embedded definition, transform it to special parsing + $c =~ s/^UNIQUE KEY/INDEX UNIQUE/is; + # Remove things that are not possible with postgres + $c =~ s/(PRIMARY KEY.*)NOT NULL/$1/is; + # Rewrite some parts for easiest/generic parsing + my $tbn = $tb_name; + $tbn =~ s/\./_/gs; + $c =~ s/^(PRIMARY KEY|UNIQUE)/CONSTRAINT ora2pg_ukey_$tbn $1/is; + $c =~ s/^(CHECK[^,;]+)DEFERRABLE\s+INITIALLY\s+DEFERRED/$1/is; + $c =~ s/^CHECK\b/CONSTRAINT ora2pg_ckey_$tbn CHECK/is; + $c =~ s/^FOREIGN KEY/CONSTRAINT ora2pg_fkey_$tbn FOREIGN KEY/is; + + if (!$self->{preserve_case}) { + $c =~ s/"//gs; + } + $c =~ s/\(\s+/\(/gs; + + # Get column name + if ($c =~ s/^\s*([^\s]+)\s*//s) + { + my $c_name = $1; + $c_name =~ s/"//g; + # Retrieve all columns information + if (!grep(/^\Q$c_name\E$/i, 'CONSTRAINT','INDEX')) + { + $cur_c_name = $c_name; + $c_name =~ s/\./_/gs; + my $c_default = ''; + my $virt_col = 'NO'; + $c =~ s/\s+ENABLE//is; + if ($c =~ s/\bGENERATED\s+(ALWAYS|BY\s+DEFAULT)\s+(ON\s+NULL\s+)?AS\s+IDENTITY\s*(.*)//is) + { + $self->{identity_info}{$tb_name}{$c_name}{generation} = $1; + my $options = $3; + $self->{identity_info}{$tb_name}{$c_name}{options} = $3; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(SCALE|EXTEND|SESSION)_FLAG: .//isg; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/KEEP_VALUE: .//is; + + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(START WITH):/$1/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(INCREMENT BY):/$1/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/MAX_VALUE:/MAXVALUE/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/MIN_VALUE:/MINVALUE/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CYCLE_FLAG: N/NO CYCLE/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/NOCYCLE/NO CYCLE/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CYCLE_FLAG: Y/CYCLE/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CACHE_SIZE:/CACHE/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CACHE_SIZE:/CACHE/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/ORDER_FLAG: .//is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/,//gs; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s$//s; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/CACHE\s+0/CACHE 1/is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOORDER//is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOKEEP//is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOSCALE//is; + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s*NOT\s+NULL//is; + # Be sure that we don't exceed the bigint max value, + # we assume that the increment is always positive + if ($self->{identity_info}{$tb_name}{$c_name}{options} =~ /MAXVALUE\s+(\d+)/is) { + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/(MAXVALUE)\s+\d+/$1 9223372036854775807/is; + } + $self->{identity_info}{$tb_name}{$c_name}{options} =~ s/\s+/ /igs; + } + elsif ($c =~ s/\b(GENERATED ALWAYS AS|AS)\s+(.*)//is) + { + $virt_col = 'YES'; + $c_default = $2; + $c_default =~ s/\s+VIRTUAL//is; + } + my $c_type = ''; + if ($c =~ s/^ENUM\s*(\([^\(\)]+\))\s*//is) + { + $c_type = 'varchar'; + my $ck_name = 'ora2pg_ckey_' . $c_name; + $self->_parse_constraint($tb_name, $c_name, "$ck_name CHECK ($c_name IN $1)"); + } elsif ($c =~ s/^([^\s\(]+)\s*//s) { + $c_type = $1; + } elsif ($c_default) + { + # Try to guess a type the virtual column was declared without one, + # but always default to text and always display a warning. + if ($c_default =~ /ROUND\s*\(/is) { + $c_type = 'numeric'; + } elsif ($c_default =~ /TO_DATE\s\(/is) { + $c_type = 'timestamp'; + } else { + $c_type = 'text'; + } + print STDERR "WARNING: Virtual column $tb_name.$cur_c_name has no data type defined, using $c_type but you need to check that this is the right type.\n"; + } + else + { + next; + } + my $c_length = ''; + my $c_scale = ''; + if ($c =~ s/^\(([^\)]+)\)\s*//s) + { + $c_length = $1; + if ($c_length =~ s/\s*,\s*(\d+)\s*//s) { + $c_scale = $1; + } + } + my $c_nullable = 1; + if ($c =~ s/CONSTRAINT\s*([^\s]+)?\s*NOT NULL//s) { + $c_nullable = 0; + } elsif ($c =~ s/NOT NULL//) { + $c_nullable = 0; + } + + if (($c =~ s/(UNIQUE|PRIMARY KEY)\s*\(([^\)]+)\)//is) || ($c =~ s/(UNIQUE|PRIMARY KEY)\s*//is)) + { + $c_name =~ s/\./_/gs; + my $pk_name = 'ora2pg_ukey_' . $c_name; + my $cols = $c_name; + if ($2) { + $cols = $2; + } + $self->_parse_constraint($tb_name, $c_name, "$pk_name $1 ($cols)"); + + } + elsif ( ($c =~ s/CONSTRAINT\s([^\s]+)\sCHECK\s*\(([^\)]+)\)//is) || ($c =~ s/CHECK\s*\(([^\)]+)\)//is) ) + { + $c_name =~ s/\./_/gs; + my $pk_name = 'ora2pg_ckey_' . $c_name; + my $chk_search = $1; + if ($2) + { + $pk_name = $1; + $chk_search = $2; + } + $chk_search .= $c if ($c eq ')'); + $self->_parse_constraint($tb_name, $c_name, "$pk_name CHECK ($chk_search)"); + } + elsif ($c =~ s/REFERENCES\s+([^\(\s]+)\s*\(([^\)]+)\)//is) + { + + $c_name =~ s/\./_/gs; + my $pk_name = 'ora2pg_fkey_' . $c_name; + my $chk_search = $1 . "($2)"; + $chk_search =~ s/\s+//gs; + $self->_parse_constraint($tb_name, $c_name, "$pk_name FOREIGN KEY ($c_name) REFERENCES $chk_search"); + } + + my $auto_incr = 0; + if ($c =~ s/\s*AUTO_INCREMENT\s*//is) { + $auto_incr = 1; + } + # At this stage only the DEFAULT part might be on the string + if ($c =~ /\bDEFAULT\s+/is) + { + if ($c =~ s/\bDEFAULT\s+('[^']+')\s*//is) { + $c_default = $1; + } elsif ($c =~ s/\bDEFAULT\s+([^\s]+)\s*$//is) { + $c_default = $1; + } elsif ($c =~ s/\bDEFAULT\s+(.*)$//is) { + $c_default = $1; + } + $c_default =~ s/"//gs; + if ($self->{plsql_pgsql}) { + $c_default = Ora2Pg::PLSQL::convert_plsql_code($self, $c_default); + } + } + if ($c_type =~ /date|timestamp/i && $c_default =~ /'0000-00-00/) + { + if ($self->{replace_zero_date}) { + $c_default = $self->{replace_zero_date}; + } else { + $c_default =~ s/^'0000-00-00/'1970-01-01/; + } + if ($c_default =~ /^[\-]*INFINITY$/) { + $c_default .= "::$c_type"; + } + } + # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE + push(@{$self->{tables}{$tb_name}{column_info}{$c_name}}, ($c_name, $c_type, $c_length, $c_nullable, $c_default, $c_length, $c_scale, $c_length, $tb_name, '', $virt_col, $pos, $auto_incr)); + } + elsif (uc($c_name) eq 'CONSTRAINT') + { + $self->_parse_constraint($tb_name, $cur_c_name, $c); + } + elsif (uc($c_name) eq 'INDEX') + { + if ($c =~ /^\s*UNIQUE\s+([^\s]+)\s+\(([^\)]+)\)/) + { + my $idx_name = $1; + my @cols = (); + push(@cols, split(/\s*,\s*/, $2)); + map { s/^"//; s/"$//; } @cols; + $self->{tables}{$tb_name}{unique_key}->{$idx_name}{type} = 'U'; + $self->{tables}{$tb_name}{unique_key}->{$idx_name}{generated} = 0; + $self->{tables}{$tb_name}{unique_key}->{$idx_name}{index_name} = $idx_name; + push(@{$self->{tables}{$tb_name}{unique_key}->{$idx_name}{columns}}, @cols); + } + elsif ($c =~ /^\s*([^\s]+)\s+\(([^\)]+)\)/) + { + my $idx_name = $1; + my @cols = (); + push(@cols, split(/\s*,\s*/, $2)); + map { s/^"//; s/"$//; } @cols; + push(@{$self->{tables}{$tb_name}{indexes}{$idx_name}}, @cols); + } + } + } + $pos++; + } + map {s/^/\t/; s/$/,\n/; } @column_defs; + # look for storage information + if ($tb_param =~ /TABLESPACE[\s]+([^\s]+)/is) { + $self->{tables}{$tb_name}{table_info}{tablespace} = $1; + $self->{tables}{$tb_name}{table_info}{tablespace} =~ s/"//gs; + } + if ($tb_param =~ /PCTFREE\s+(\d+)/is) { + # We only take care of pctfree upper than the default + if ($1 > 10) { + # fillfactor must be >= 10 + $self->{tables}{$tb_name}{table_info}{fillfactor} = 100 - min(90, $1); + } + } + if ($tb_param =~ /\bNOLOGGING\b/is) { + $self->{tables}{$tb_name}{table_info}{nologging} = 1; + } + + if ($tb_param =~ /ORGANIZATION EXTERNAL/is) { + if ($tb_param =~ /DEFAULT DIRECTORY ([^\s]+)/is) { + $self->{external_table}{$tb_name}{director} = $1; + } + $self->{external_table}{$tb_name}{delimiter} = ','; + if ($tb_param =~ /FIELDS TERMINATED BY '(.)'/is) { + $self->{external_table}{$tb_name}{delimiter} = $1; + } + if ($tb_param =~ /PREPROCESSOR EXECDIR\s*:\s*'([^']+)'/is) { + $self->{external_table}{$tb_name}{program} = $1; + } + if ($tb_param =~ /LOCATION\s*\(\s*'([^']+)'\s*\)/is) { + $self->{external_table}{$tb_name}{location} = $1; + } + } + + } elsif ($content =~ s/CREATE\s+(UNIQUE|BITMAP)?\s*INDEX\s+([^\s]+)\s+ON\s+([^\s\(]+)\s*\((.*)\)//is) { + my $is_unique = $1; + my $idx_name = $2; + my $tb_name = $3; + my $idx_def = $4; + $idx_name =~ s/"//gs; + $tb_name =~ s/\s+/ /gs; + $idx_def =~ s/\s+/ /gs; + $idx_def =~ s/\s*nologging//is; + $idx_def =~ s/STORAGE\s*\([^\)]+\)\s*//is; + $idx_def =~ s/COMPRESS(\s+\d+)?\s*//is; + # look for storage information + if ($idx_def =~ s/TABLESPACE\s*([^\s]+)\s*//is) { + $self->{tables}{$tb_name}{idx_tbsp}{$idx_name} = $1; + $self->{tables}{$tb_name}{idx_tbsp}{$idx_name} =~ s/"//gs; + } + if ($idx_def =~ s/ONLINE\s*//is) { + $self->{tables}{$tb_name}{concurrently}{$idx_name} = 1; + } + if ($idx_def =~ s/INDEXTYPE\s+IS\s+.*SPATIAL_INDEX//is) { + $self->{tables}{$tb_name}{spatial}{$idx_name} = 1; + $self->{tables}{$tb_name}{idx_type}{$idx_name}{type} = 'SPATIAL INDEX'; + $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_name} = 'SPATIAL_INDEX'; + } + if ($idx_def =~ s/layer_gtype=([^\s,]+)//is) { + $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_constraint} = uc($1); + } + if ($idx_def =~ s/sdo_indx_dims=(\d)//is) { + $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_dims} = $1; + } + $idx_def =~ s/\)[^\)]*$//s; + if ($is_unique eq 'BITMAP') { + $is_unique = ''; + $self->{tables}{$tb_name}{idx_type}{$idx_name}{type_name} = 'BITMAP'; + } + $self->{tables}{$tb_name}{uniqueness}{$idx_name} = $is_unique || ''; + $idx_def =~ s/SYS_EXTRACT_UTC\s*\(([^\)]+)\)/$1/isg; + push(@{$self->{tables}{$tb_name}{indexes}{$idx_name}}, $idx_def); + $self->{tables}{$tb_name}{idx_type}{$idx_name}{type} = 'NORMAL'; + if ($idx_def =~ /\(/s) { + $self->{tables}{$tb_name}{idx_type}{$idx_name}{type} = 'FUNCTION-BASED'; + } + + if (!exists $self->{tables}{$tb_name}{table_info}{type}) { + $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; + $self->{tables}{$tb_name}{table_info}{num_rows} = 0; + $tid++; + $self->{tables}{$tb_name}{internal_id} = $tid; + } + + } elsif ($content =~ s/ALTER\s+TABLE\s+([^\s]+)\s+ADD\s*\(*\s*(.*)//is) { + my $tb_name = $1; + $tb_name =~ s/"//g; + my $tb_def = $2; + # Oracle allow multiple constraints declaration inside a single ALTER TABLE + while ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+CHECK\s*(\(.*?\))\s+(ENABLE|DISABLE|VALIDATE|NOVALIDATE|DEFERRABLE|INITIALLY|DEFERRED|USING\s+INDEX|\s+)+([^,]*)//is) { + my $constname = $1; + my $code = $2; + my $states = $3; + my $tbspace_move = $4; + if (!exists $self->{tables}{$tb_name}{table_info}{type}) { + $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; + $self->{tables}{$tb_name}{table_info}{num_rows} = 0; + $tid++; + $self->{tables}{$tb_name}{internal_id} = $tid; + } + my $validate = ''; + $validate = ' NOT VALID' if ( $states =~ /NOVALIDATE/is); + push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT \L$constname\E CHECK $code$validate"); + if ( $tbspace_move =~ /USING\s+INDEX\s+TABLESPACE\s+([^\s]+)/is) { + if ($self->{use_tablespace}) { + $tbspace_move = "ALTER INDEX $constname SET TABLESPACE " . lc($1); + push(@{$self->{tables}{$tb_name}{alter_index}}, $tbspace_move); + } + } elsif ($tbspace_move =~ /USING\s+INDEX\s+([^\s]+)/is) { + $self->{tables}{$tb_name}{alter_table}[-1] .= " USING INDEX " . lc($1); + } + + } + while ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+FOREIGN\s+KEY\s*(\(.*?\)\s+REFERENCES\s+[^\s]+\s*\(.*?\))\s*([^,\)]+|$)//is) { + my $constname = $1; + my $other_def = $3; + if (!exists $self->{tables}{$tb_name}{table_info}{type}) { + $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; + $self->{tables}{$tb_name}{table_info}{num_rows} = 0; + $tid++; + $self->{tables}{$tb_name}{internal_id} = $tid; + } + push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD CONSTRAINT \L$constname\E FOREIGN KEY $2"); + if ($other_def =~ /(ON\s+DELETE\s+(?:NO ACTION|RESTRICT|CASCADE|SET NULL))/is) { + $self->{tables}{$tb_name}{alter_table}[-1] .= " $1"; + } + if ($other_def =~ /(ON\s+UPDATE\s+(?:NO ACTION|RESTRICT|CASCADE|SET NULL))/is) { + $self->{tables}{$tb_name}{alter_table}[-1] .= " $1"; + } + my $validate = ''; + $validate = ' NOT VALID' if ( $other_def =~ /NOVALIDATE/is); + $self->{tables}{$tb_name}{alter_table}[-1] .= $validate; + } + # We can just have one primary key constraint + if ($tb_def =~ s/CONSTRAINT\s+([^\s]+)\s+PRIMARY KEY//is) { + my $constname = lc($1); + $tb_def =~ s/^[^\(]+//; + if ( $tb_def =~ s/USING\s+INDEX\s+TABLESPACE\s+([^\s]+).*//s) { + $tb_def =~ s/\s+$//; + if ($self->{use_tablespace}) { + my $tbspace_move = "ALTER INDEX $constname SET TABLESPACE $1"; + push(@{$self->{tables}{$tb_name}{alter_index}}, $tbspace_move); + } + push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD PRIMARY KEY $constname " . lc($tb_def)); + } elsif ($tb_def =~ s/USING\s+INDEX\s+([^\s]+).*//s) { + push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD PRIMARY KEY " . lc($tb_def)); + $self->{tables}{$tb_name}{alter_table}[-1] .= " USING INDEX " . lc($1); + } elsif ($tb_def) { + push(@{$self->{tables}{$tb_name}{alter_table}}, "ADD PRIMARY KEY $constname " . lc($tb_def)); + } + if (!exists $self->{tables}{$tb_name}{table_info}{type}) { + $self->{tables}{$tb_name}{table_info}{type} = 'TABLE'; + $self->{tables}{$tb_name}{table_info}{num_rows} = 0; + $tid++; + $self->{tables}{$tb_name}{internal_id} = $tid; + } + } + } + + } + + # Extract comments + $self->read_comment_from_file(); +} + +sub read_comment_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + my $tid = 0; + + while ($content =~ s/COMMENT\s+ON\s+TABLE\s+([^\s]+)\s+IS\s+'([^;]+);//is) { + my $tb_name = $1; + my $tb_comment = $2; + $tb_name =~ s/"//g; + $tb_comment =~ s/'\s*$//g; + if (exists $self->{tables}{$tb_name}) { + $self->{tables}{$tb_name}{table_info}{comment} = $tb_comment; + } + } + + while ($content =~ s/COMMENT\s+ON\s+COLUMN\s+([^\s]+)\s+IS\s+'([^;]+);//is) { + my $tb_name = $1; + my $tb_comment = $2; + $tb_name =~ s/"//g; + $tb_comment =~ s/'\s*$//g; + if ($tb_name =~ s/\.([^\.]+)$//) { + if (exists $self->{tables}{$tb_name}) { + $self->{tables}{$tb_name}{column_comments}{"\L$1\E"} = $tb_comment; + } elsif (exists $self->{views}{$tb_name}) { + $self->{views}{$tb_name}{column_comments}{"\L$1\E"} = $tb_comment; + } + } + } + +} + +sub read_view_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Clear content from comment and text constant for better parsing + $self->_remove_comments(\$content); + + my $tid = 0; + + $content =~ s/\s+NO\s+FORCE\s+/ /gs; + $content =~ s/\s+FORCE\s+/ /gs; + $content =~ s/\s+OR\s+REPLACE\s+/ /gs; + $content =~ s/CREATE\s+VIEW\s+([^\s]+)\s+OF\s+(.*?)\s+AS\s+/CREATE VIEW $1 AS /sg; + # Views with aliases + while ($content =~ s/CREATE\s+VIEW\s+([^\s]+)\s*\((.*?)\)\s+AS\s+([^;]+)(;|$)//is) { + my $v_name = $1; + my $v_alias = $2; + my $v_def = $3; + $v_name =~ s/"//g; + $tid++; + $self->{views}{$v_name}{text} = $v_def; + $self->{views}{$v_name}{iter} = $tid; + # Remove constraint + while ($v_alias =~ s/(,[^,\(]+\(.*)$//) {}; + my @aliases = split(/\s*,\s*/, $v_alias); + foreach (@aliases) { + s/^\s+//; + s/\s+$//; + my @tmp = split(/\s+/); + push(@{$self->{views}{$v_name}{alias}}, \@tmp); + } + } + # Standard views + while ($content =~ s/CREATE\sVIEW[\s]+([^\s]+)\s+AS\s+([^;]+);//i) { + my $v_name = $1; + my $v_def = $2; + $v_name =~ s/"//g; + $tid++; + $self->{views}{$v_name}{text} = $v_def; + } + + # Extract comments + $self->read_comment_from_file(); +} + +sub read_grant_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Clear content from comment and text constant for better parsing + $self->_remove_comments(\$content); + + my $tid = 0; + + # Extract grant information + while ($content =~ s/GRANT\s+(.*?)\s+ON\s+([^\s]+)\s+TO\s+([^;]+)(\s+WITH GRANT OPTION)?;//i) { + my $g_priv = $1; + my $g_name = $2; + $g_name =~ s/"//g; + my $g_user = $3; + my $g_option = $4; + $g_priv =~ s/\s+//g; + $tid++; + $self->{grants}{$g_name}{type} = ''; + push(@{$self->{grants}{$g_name}{privilege}{$g_user}}, split(/,/, $g_priv)); + if ($g_priv =~ /EXECUTE/) { + $self->{grants}{$table}{type} = 'PACKAGE BODY'; + } else { + $self->{grants}{$table}{type} = 'TABLE'; + } + } + +} + +sub read_trigger_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Clear content from comment and text constant for better parsing + $self->_remove_comments(\$content); + + my $tid = 0; + my $doloop = 1; + my @triggers_decl = split(/(?:CREATE)?(?:\s+OR\s+REPLACE)?\s*(?:DEFINER=[^\s]+)?\s*\bTRIGGER(\s+|$)/is, $content); + foreach $content (@triggers_decl) + { + my $t_name = ''; + my $t_pos = ''; + my $t_event = ''; + my $tb_name = ''; + my $trigger = ''; + my $t_type = ''; + if ($content =~ s/^([^\s]+)\s+(BEFORE|AFTER|INSTEAD\s+OF)\s+(.*?)\s+ON\s+([^\s]+)\s+(.*)(\bEND\s*(?!IF|LOOP|CASE|INTO|FROM|,)[a-z0-9_]*(?:;|$))//is) + { + $t_name = $1; + $t_pos = $2; + $t_event = $3; + $tb_name = $4; + $trigger = $5 . $6; + $t_name =~ s/"//g; + } + elsif ($content =~ s/^([^\s]+)\s+(BEFORE|AFTER|INSTEAD|\s+|OF)((?:INSERT|UPDATE|DELETE|OR|\s+|OF)+\s+(?:.*?))*\s+ON\s+([^\s]+)\s+(.*)(\bEND\s*(?!IF|LOOP|CASE|INTO|FROM|,)[a-z0-9_]*(?:;|$))//is) + { + $t_name = $1; + $t_pos = $2; + $t_event = $3; + $tb_name = $4; + $trigger = $5 . $6; + $t_name =~ s/"//g; + } + + next if (!$t_name || ! $tb_name); + + # Remove referencing clause, not supported by PostgreSQL + $trigger =~ s/REFERENCING\s+(.*?)(FOR\s+EACH\s+)/$2/is; + + if ($trigger =~ s/^\s*(FOR\s+EACH\s+)(ROW|STATEMENT)\s*//is) { + $t_type = $1 . $2; + } + my $t_when_cond = ''; + if ($trigger =~ s/^\s*WHEN\s+(.*?)\s+((?:BEGIN|DECLARE|CALL).*)//is) + { + $t_when_cond = $1; + $trigger = $2; + if ($trigger =~ /^(BEGIN|DECLARE)/i) { + ($trigger, $content) = &_get_plsql_code($trigger); + } + else + { + $trigger =~ s/([^;]+;)\s*(.*)/$1/; + $content = $2; + } + } + else + { + if ($trigger =~ /^(BEGIN|DECLARE)/i) { + ($trigger, $content) = &_get_plsql_code($trigger); + } + } + $tid++; + + # TRIGGER_NAME, TRIGGER_TYPE, TRIGGERING_EVENT, TABLE_NAME, TRIGGER_BODY, WHEN_CLAUSE, DESCRIPTION,ACTION_TYPE + $trigger =~ s/\bEND\s+[^\s]+\s+$/END/is; + my $when_event = ''; + if ($t_when_cond) { + $when_event = "$t_name\n$t_pos $t_event ON $tb_name\n$t_type"; + } + push(@{$self->{triggers}}, [($t_name, $t_pos, $t_event, $tb_name, $trigger, $t_when_cond, $when_event, $t_type)]); + } +} + +sub read_sequence_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Clear content from comment and text constant for better parsing + $self->_remove_comments(\$content, 1); + $content =~ s/\%ORA2PG_COMMENT\d+\%//gs; + my $tid = 0; + + # Sequences + while ($content =~ s/CREATE\s+SEQUENCE[\s]+([^\s;]+)\s*([^;]+);//i) + { + my $s_name = $1; + my $s_def = $2; + $s_name =~ s/"//g; + $s_def =~ s/\s+/ /g; + $tid++; + my @seq_info = (); + + # Field of @seq_info + # SEQUENCE_NAME, MIN_VALUE, MAX_VALUE, INCREMENT_BY, LAST_NUMBER, CACHE_SIZE, CYCLE_FLAG, SEQUENCE_OWNER FROM $self->{prefix}_SEQUENCES"; + push(@seq_info, $s_name); + if ($s_def =~ /MINVALUE\s+([\-\d]+)/i) { + push(@seq_info, $1); + } else { + push(@seq_info, ''); + } + if ($s_def =~ /MAXVALUE\s+([\-\d]+)/i) + { + if ($1 > 9223372036854775807) { + push(@seq_info, 9223372036854775807); + } else { + push(@seq_info, $1); + } + } else { + push(@seq_info, ''); + } + if ($s_def =~ /INCREMENT\s*(?:BY)?\s+([\-\d]+)/i) { + push(@seq_info, $1); + } else { + push(@seq_info, 1); + } + + if ($s_def =~ /START\s+WITH\s+([\-\d]+)/i) { + push(@seq_info, $1); + } else { + push(@seq_info, ''); + } + if ($s_def =~ /CACHE\s+(\d+)/i) { + push(@seq_info, $1); + } else { + push(@seq_info, ''); + } + if ($s_def =~ /NOCYCLE/i) { + push(@seq_info, 'NO'); + } else { + push(@seq_info, 'YES'); + } + if ($s_name =~ /^([^\.]+)\./i) { + push(@seq_info, $1); + } else { + push(@seq_info, ''); + } + push(@{$self->{sequences}}, \@seq_info); + } +} + +sub read_tablespace_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + my @tbsps = split(/\s*;\s*/, $content); + # tablespace without undo ones + foreach $content (@tbsps) { + $content .= ';'; + if ($content =~ /CREATE\s+(?:BIGFILE|SMALLFILE)?\s*(?:TEMPORARY)?\s*TABLESPACE\s+([^\s;]+)\s*([^;]*);/is) { + my $t_name = $1; + my $t_def = $2; + $t_name =~ s/"//g; + if ($t_def =~ /(?:DATA|TEMP)FILE\s+'([^']+)'/is) { + my $t_path = $1; + $t_path =~ s/:/\//g; + $t_path =~ s/\\/\//g; + if (dirname($t_path) eq '.') { + $t_path = 'change_tablespace_dir'; + } else { + $t_path = dirname($t_path); + } + # TYPE - TABLESPACE_NAME - FILEPATH - OBJECT_NAME + @{$self->{tablespaces}{TABLE}{$t_name}{$t_path}} = (); + } + + } + } +} + +sub read_directory_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Directory + while ($content =~ s/CREATE(?: OR REPLACE)?\s+DIRECTORY\s+([^\s]+)\s+AS\s+'([^']+)'\s*;//is) { + my $d_name = uc($1); + my $d_def = $2; + $d_name =~ s/"//g; + if ($d_def !~ /\/$/) { + $d_def .= '/'; + } + $self->{directory}{$d_name}{path} = $d_def; + } + + # Directory + while ($content =~ s/GRANT\s+(.*?)ON\s+DIRECTORY\s+([^\s]+)\s+TO\s+([^;\s]+)\s*;//is) { + my $d_grant = $1; + my $d_name = uc($2); + my $d_user = uc($3); + $d_name =~ s/"//g; + $d_user =~ s/"//g; + $self->{directory}{$d_name}{grantee}{$d_user} = $d_grant; + } +} + +sub read_synonym_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Directory + while ($content =~ s/CREATE(?: OR REPLACE)?(?: PUBLIC)?\s+SYNONYM\s+([^\s]+)\s+FOR\s+([^;\s]+)\s*;//is) { + my $s_name = uc($1); + my $s_def = $2; + $s_name =~ s/"//g; + $s_def =~ s/"//g; + if ($s_name =~ s/^([^\.]+)\.//) { + $self->{synonyms}{$s_name}{owner} = $1; + } else { + $self->{synonyms}{$s_name}{owner} = $self->{schema}; + } + if ($s_def =~ s/@(.*)//) { + $self->{synonyms}{$s_name}{dblink} = $1; + } + if ($s_def =~ s/^([^\.]+)\.//) { + $self->{synonyms}{$s_name}{table_owner} = $1; + } + $self->{synonyms}{$s_name}{table_name} = $s_def; + } + +} + +sub read_dblink_from_file +{ + my $self = shift; + + # Load file in a single string + my $content = $self->_get_dml_from_file(); + + # Directory + while ($content =~ s/CREATE(?: SHARED)?(?: PUBLIC)?\s+DATABASE\s+LINK\s+([^\s]+)\s+CONNECT TO\s+([^\s]+)\s*([^;]+);//is) { + my $d_name = $1; + my $d_user = $2; + my $d_auth = $3; + $d_name =~ s/"//g; + $d_user =~ s/"//g; + $self->{dblink}{$d_name}{owner} = $self->{shema}; + $self->{dblink}{$d_name}{user} = $d_user; + $self->{dblink}{$d_name}{username} = $self->{pg_user} || $d_user; + if ($d_auth =~ s/USING\s+([^\s]+)//) { + $self->{dblink}{$d_name}{host} = $1; + $self->{dblink}{$d_name}{host} =~ s/'//g; + } + if ($d_auth =~ s/IDENTIFIED\s+BY\s+([^\s]+)//) { + $self->{dblink}{$d_name}{password} = $1; + } + if ($d_auth =~ s/AUTHENTICATED\s+BY\s+([^\s]+)\s+IDENTIFIED\s+BY\s+([^\s]+)//) { + $self->{dblink}{$d_name}{user} = $1; + $self->{dblink}{$d_name}{password} = $2; + $self->{dblink}{$d_name}{username} = $self->{pg_user} || $1; + } + } + + # Directory + while ($content =~ s/CREATE(?: SHARED)?(?: PUBLIC)?\s+DATABASE\s+LINK\s+([^\s]+)\s+USING\s+([^;]+);//is) { + my $d_name = $1; + my $d_conn = $2; + $d_name =~ s/"//g; + $d_conn =~ s/'//g; + $self->{dblink}{$d_name}{owner} = $self->{shema}; + $self->{dblink}{$d_name}{host} = $d_conn; + } + + +} + + +=head2 _views + +This function is used to retrieve all views information. + +Sets the main hash of the views definition $self->{views}. +Keys are the names of all views retrieved from the current +database and values are the text definitions of the views. + +It then sets the main hash as follows: + + # Definition of the view + $self->{views}{$table}{text} = $lview_infos{$table}; + +=cut + +sub _views +{ + my ($self) = @_; + + # Get all views information + $self->logit("Retrieving views information...\n", 1); + my %view_infos = $self->_get_views(); + # Retrieve comment of each columns + my %columns_comments = $self->_column_comments(); + foreach my $view (keys %columns_comments) { + next if (!exists $view_infos{$view}); + foreach my $c (keys %{$columns_comments{$view}}) { + $self->{views}{$view}{column_comments}{$c} = $columns_comments{$view}{$c}; + } + } + + my $i = 1; + foreach my $view (sort keys %view_infos) { + $self->logit("[$i] Scanning $view...\n", 1); + $self->{views}{$view}{text} = $view_infos{$view}{text}; + $self->{views}{$view}{owner} = $view_infos{$view}{owner}; + $self->{views}{$view}{iter} = $view_infos{$view}{iter} if (exists $view_infos{$view}{iter}); + $self->{views}{$view}{comment} = $view_infos{$view}{comment}; + # Retrieve also aliases from views + $self->{views}{$view}{alias} = $view_infos{$view}{alias}; + $i++; + } + +} + +=head2 _materialized_views + +This function is used to retrieve all materialized views information. + +Sets the main hash of the views definition $self->{materialized_views}. +Keys are the names of all materialized views retrieved from the current +database and values are the text definitions of the views. + +It then sets the main hash as follows: + + # Definition of the materialized view + $self->{materialized_views}{text} = $mview_infos{$view}; + +=cut + +sub _materialized_views +{ + my ($self) = @_; + + # Get all views information + $self->logit("Retrieving materialized views information...\n", 1); + my %mview_infos = $self->_get_materialized_views(); + + my $i = 1; + foreach my $table (sort keys %mview_infos) + { + $self->logit("[$i] Scanning $table...\n", 1); + $self->{materialized_views}{$table}{text} = $mview_infos{$table}{text}; + $self->{materialized_views}{$table}{updatable}= $mview_infos{$table}{updatable}; + $self->{materialized_views}{$table}{refresh_mode}= $mview_infos{$table}{refresh_mode}; + $self->{materialized_views}{$table}{refresh_method}= $mview_infos{$table}{refresh_method}; + $self->{materialized_views}{$table}{no_index}= $mview_infos{$table}{no_index}; + $self->{materialized_views}{$table}{rewritable}= $mview_infos{$table}{rewritable}; + $self->{materialized_views}{$table}{build_mode}= $mview_infos{$table}{build_mode}; + $self->{materialized_views}{$table}{owner}= $mview_infos{$table}{owner}; + $i++; + } + + # Retrieve index informations + if (scalar keys %mview_infos) + { + my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}); + foreach my $tb (keys %{$indexes}) + { + next if (!exists $self->{materialized_views}{$tb}); + %{$self->{materialized_views}{$tb}{indexes}} = %{$indexes->{$tb}}; + } + foreach my $tb (keys %{$idx_type}) + { + next if (!exists $self->{materialized_views}{$tb}); + %{$self->{materialized_views}{$tb}{idx_type}} = %{$idx_type->{$tb}}; + } + } +} + +=head2 _tablespaces + +This function is used to retrieve all Oracle Tablespaces information. + +Sets the main hash $self->{tablespaces}. + +=cut + +sub _tablespaces +{ + my ($self) = @_; + + $self->logit("Retrieving tablespaces information...\n", 1); + $self->{tablespaces} = $self->_get_tablespaces(); + $self->{list_tablespaces} = $self->_list_tablespaces(); + +} + +=head2 _partitions + +This function is used to retrieve all Oracle partition information. + +Sets the main hash $self->{partition}. + +=cut + +sub _partitions +{ + my ($self) = @_; + + $self->logit("Retrieving partitions information...\n", 1); + ($self->{partitions}, $self->{partitions_default}) = $self->_get_partitions(); + + ($self->{subpartitions}, $self->{subpartitions_default}) = $self->_get_subpartitions(); + + # Get partition list meta information + %{ $self->{partitions_list} } = $self->_get_partitioned_table(); + %{ $self->{subpartitions_list} } = $self->_get_subpartitioned_table(); + + # Look for main table indexes to reproduce them on partition + my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}, 0); + foreach my $tb (keys %{$indexes}) { + %{$self->{tables}{$tb}{indexes}} = %{$indexes->{$tb}}; + } + foreach my $tb (keys %{$idx_type}) { + %{$self->{tables}{$tb}{idx_type}} = %{$idx_type->{$tb}}; + } + foreach my $tb (keys %{$idx_tbsp}) { + %{$self->{tables}{$tb}{idx_tbsp}} = %{$idx_tbsp->{$tb}}; + } + foreach my $tb (keys %{$uniqueness}) { + %{$self->{tables}{$tb}{uniqueness}} = %{$uniqueness->{$tb}}; + } + + # Retrieve all unique keys informations + my %unique_keys = $self->_unique_key('',$self->{schema}); + foreach my $tb (keys %unique_keys) { + foreach my $c (keys %{$unique_keys{$tb}}) { + $self->{tables}{$tb}{unique_key}{$c} = $unique_keys{$tb}{$c}; + } + } +} + +=head2 _dblinks + +This function is used to retrieve all Oracle dblinks information. + +Sets the main hash $self->{dblink}. + +=cut + +sub _dblinks +{ + my ($self) = @_; + + $self->logit("Retrieving dblinks information...\n", 1); + %{$self->{dblink}} = $self->_get_dblink(); + +} + +=head2 _directories + +This function is used to retrieve all Oracle directories information. + +Sets the main hash $self->{directory}. + +=cut + +sub _directories +{ + my ($self) = @_; + + $self->logit("Retrieving directories information...\n", 1); + %{$self->{directory}} = $self->_get_directory(); + +} + + +sub get_replaced_tbname +{ + my ($self, $tmptb) = @_; + + if (exists $self->{replaced_tables}{"\L$tmptb\E"} && $self->{replaced_tables}{"\L$tmptb\E"}) { + $self->logit("\tReplacing table $tmptb as " . $self->{replaced_tables}{lc($tmptb)} . "...\n", 1); + $tmptb = $self->{replaced_tables}{lc($tmptb)}; + } + + $tmptb = $self->quote_object_name($tmptb); + + return $tmptb; +} + +sub get_tbname_with_suffix +{ + my ($self, $tmptb, $suffix) = @_; + + return $self->quote_object_name($tmptb . $suffix); +} + + +sub _export_table_data +{ + my ($self, $table, $dirprefix, $sql_header) = @_; + + # Rename table and double-quote it if required + my $tmptb = $self->get_replaced_tbname($table); + + # Open output file + $self->data_dump($sql_header, $table) if (!$self->{pg_dsn} && $self->{file_per_table}); + + my $total_record = 0; + + # When copy freeze is required, force a transaction with a truncate + if ($self->{copy_freeze} && !$self->{pg_dsn}) { + $self->{truncate_table} = 1; + if ($self->{file_per_table}) { + $self->data_dump("BEGIN;\n", $table); + } else { + $self->dump("\nBEGIN;\n"); + } + } else { + $self->{copy_freeze} = ''; + } + + # Open a new connection to PostgreSQL destination with parallel table export + my $local_dbh = undef; + if (($self->{parallel_tables} > 1) && $self->{pg_dsn}) { + $local_dbh = $self->_send_to_pgdb(); + } else { + $local_dbh = $self->{dbhdest}; + } + + if ($self->{global_delete} || exists $self->{delete}{"\L$table\E"}) + { + my $delete_clause = ''; + my $delete_clause_start = "DELETE"; + if ($self->{datadiff}) { + $delete_clause_start = "INSERT INTO " . $self->get_tbname_with_suffix($tmptb, $self->{datadiff_del_suffix}) . " SELECT *"; + } + if (exists $self->{delete}{"\L$table\E"} && $self->{delete}{"\L$table\E"}) { + $delete_clause = "$delete_clause_start FROM $tmptb WHERE " . $self->{delete}{"\L$table\E"} . ";"; + $self->logit("\tApplying DELETE clause on table: " . $self->{delete}{"\L$table\E"} . "\n", 1); + } elsif ($self->{global_delete}) { + $delete_clause = "$delete_clause_start FROM $tmptb WHERE " . $self->{global_delete} . ";"; + $self->logit("\tApplying DELETE global clause: " . $self->{global_delete} . "\n", 1); + + } + if ($delete_clause) { + if ($self->{pg_dsn}) { + $self->logit("Deleting from table $table...\n", 1); + my $s = $local_dbh->do("$delete_clause") or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); + } else { + if ($self->{file_per_table}) { + $self->data_dump("$delete_clause\n", $table); + } else { + $self->dump("\n$delete_clause\n"); + } + } + } + } + + # Add table truncate order if there's no global DELETE clause or one specific to the current table + if ($self->{truncate_table} && !$self->{global_delete} && !exists $self->{delete}{"\L$table\E"}) { + # Set search path + my $search_path = $self->set_search_path(); + if ($self->{pg_dsn} && !$self->{oracle_speed}) { + if ($search_path) { + $local_dbh->do($search_path) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); + } + $self->logit("Truncating table $table...\n", 1); + my $s = $local_dbh->do("TRUNCATE TABLE $tmptb;") or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); + } else { + my $head = "SET client_encoding TO '\U$self->{client_encoding}\E';\n"; + $head .= "SET synchronous_commit TO off;\n" if (!$self->{synchronous_commit}); + if ($self->{file_per_table}) { + $self->data_dump("$head$search_path\nTRUNCATE TABLE $tmptb;\n", $table); + } else { + $self->dump("\n$head$search_path\nTRUNCATE TABLE $tmptb;\n"); + } + } + } + + # With partitioned table, load data direct from table partition + if (exists $self->{partitions}{$table}) + { + foreach my $pos (sort {$self->{partitions}{$table}{$a} <=> $self->{partitions}{$table}{$b}} keys %{$self->{partitions}{$table}}) + { + my $part_name = $self->{partitions}{$table}{$pos}{name}; + my $tbpart_name = $part_name; + $tbpart_name = $table . '_' . $part_name if ($self->{prefix_partition}); + next if ($self->{allow_partition} && !grep($_ =~ /^$tbpart_name$/i, @{$self->{allow_partition}})); + + if (exists $self->{subpartitions}{$table}{$part_name}) + { + foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part_name}}) + { + my $subpart = $self->{subpartitions}{$table}{$part_name}{$p}{name}; + next if ($self->{allow_partition} && !grep($_ =~ /^$subpart$/i, @{$self->{allow_partition}})); + my $sub_tb_name = $subpart; + $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any + $sub_tb_name = "${table}_$sub_tb_name" if ($self->{prefix_partition}); + if ($self->{file_per_table} && !$self->{pg_dsn}) { + # Do not dump data again if the file already exists + next if ($self->file_exists("$dirprefix${sub_tb_name}_$self->{output}")); + } + + $self->logit("Dumping sub partition table $table ($subpart)...\n", 1); + $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $subpart, 1); + # Rename temporary filename into final name + $self->rename_dump_partfile($dirprefix, $sub_tb_name); + } + # Now load content of the default subpartition table + if ($self->{subpartitions_default}{$table}{$part_name}) + { + if (!$self->{allow_partition} || grep($_ =~ /^$self->{subpartitions_default}{$table}{$part_name}$/i, @{$self->{allow_partition}})) + { + if ($self->{file_per_table} && !$self->{pg_dsn}) + { + # Do not dump data again if the file already exists + if (!$self->file_exists("$dirprefix$self->{subpartitions_default}{$table}{$part_name}_$self->{output}")) + { + $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{subpartitions_default}{$table}{$part_name}, 1); + } + } + else + { + $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{subpartitions_default}{$table}{$part_name}, 1); + } + } + # Rename temporary filename into final name + $self->rename_dump_partfile($dirprefix, $self->{subpartitions_default}{$table}{$part_name}, $table); + } + } + else + { + if ($self->{file_per_table} && !$self->{pg_dsn}) + { + # Do not dump data again if the file already exists + next if ($self->file_exists("$dirprefix${tbpart_name}_$self->{output}")); + } + + $self->logit("Dumping partition table $table ($part_name)...\n", 1); + $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $part_name); + # Rename temporary filename into final name + $self->rename_dump_partfile($dirprefix, $part_name, $table); + } + } + # Now load content of the default partition table + if ($self->{partitions_default}{$table}) + { + if (!$self->{allow_partition} || grep($_ =~ /^$self->{partitions_default}{$table}$/i, @{$self->{allow_partition}})) + { + if ($self->{file_per_table} && !$self->{pg_dsn}) + { + # Do not dump data again if the file already exists + if (!$self->file_exists("$dirprefix$self->{partitions_default}{$table}_$self->{output}")) + { + $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{partitions_default}{$table}); + } + } + else + { + $total_record = $self->_dump_table($dirprefix, $sql_header, $table, $self->{partitions_default}{$table}); + } + # Rename temporary filename into final name + $self->rename_dump_partfile($dirprefix, $self->{partitions_default}{$table}, $table); + } + } + } + else + { + + $total_record = $self->_dump_table($dirprefix, $sql_header, $table); + } + + # When copy freeze is required, close the transaction + if ($self->{copy_freeze} && !$self->{pg_dsn}) + { + if ($self->{file_per_table}) { + $self->data_dump("COMMIT;\n", $table); + } else { + $self->dump("\nCOMMIT;\n"); + } + } + + # close the connection with parallel table export + if (($self->{parallel_tables} > 1) && $self->{pg_dsn}) { + $local_dbh->disconnect() if (defined $local_dbh); + } + + # Rename temporary filename into final name + $self->rename_dump_partfile($dirprefix, $table) if (!$self->{oracle_speed}); + + return $total_record; +} + +sub rename_dump_partfile +{ + my ($self, $dirprefix, $partname, $tbl) = @_; + + my $filename = "${dirprefix}tmp_${partname}_$self->{output}"; + my $filedest = "${dirprefix}${partname}_$self->{output}"; + if ($tbl && $self->{prefix_partition}) { + $filename = "${dirprefix}tmp_${tbl}_${partname}_$self->{output}"; + $filedest = "${dirprefix}${tbl}_${partname}_$self->{output}"; + } + if (-e $filename) { + $self->logit("Renaming temporary file $filename into $filedest\n", 1); + rename($filename, $filedest); + } +} + +sub set_refresh_count +{ + my $count = shift; + + return 500 if ($count > 10000); + return 100 if ($count > 1000); + return 10 if ($count > 100); + return 1; +} + +sub translate_function +{ + my ($self, $i, $num_total_function, %functions) = @_; + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + + # Clear memory in multiprocess mode + if ($self->{jobs} > 1) { + $self->{functions} = (); + $self->{procedures} = (); + } + + my $t0 = Benchmark->new; + + my $sql_output = ''; + my $lsize = 0; + my $lcost = 0; + my $fct_count = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_function); + foreach my $fct (sort keys %functions) + { + if (!$self->{quiet} && !$self->{debug} && ($fct_count % $PGBAR_REFRESH) == 0) + { + print STDERR $self->progress_bar($i+1, $num_total_function, 25, '=', 'functions', "generating $fct" ), "\r"; + } + $fct_count++; + $self->logit("Dumping function $fct...\n", 1); + if ($self->{file_per_function}) { + my $f = "$dirprefix${fct}_$self->{output}"; + $f =~ s/\.(?:gz|bz2)$//i; + $self->dump("\\i$self->{psql_relative_path} $f\n"); + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($fct), "$dirprefix${fct}_$self->{output}"); + } else { + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($fct), "$dirprefix$self->{output}"); + } + + my $fhdl = undef; + + $self->_remove_comments(\$functions{$fct}{text}); + $lsize = length($functions{$fct}{text}); + + if ($self->{file_per_function}) + { + $self->logit("Dumping to one file per function : ${fct}_$self->{output}\n", 1); + $fhdl = $self->open_export_file("${fct}_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + } + if ($self->{plsql_pgsql}) + { + my $sql_f = ''; + if ($self->{is_mysql}) { + $sql_f = $self->_convert_function($functions{$fct}{owner}, $functions{$fct}{text}, $fct); + } else { + $sql_f = $self->_convert_function($functions{$fct}{owner}, $functions{$fct}{text}); + } + if ( $sql_f ) + { + $sql_output .= $sql_f . "\n\n"; + if ($self->{estimate_cost}) + { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $sql_f); + $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'FUNCTION'}; + $lcost += $cost; + $self->logit("Function ${fct} estimated cost: $cost\n", 1); + $sql_output .= "-- Function ${fct} estimated cost: $cost\n"; + foreach (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) + { + next if (!$cost_detail{$_}); + $sql_output .= "\t-- $_ => $cost_detail{$_}"; + if (!$self->{is_mysql}) { + $sql_output .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_SCORE{$_}); + } else { + $sql_output .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_}); + } + $sql_output .= "\n"; + } + if ($self->{jobs} > 1) + { + my $tfh = $self->append_export_file($dirprefix . 'temp_cost_file.dat', 1); + flock($tfh, 2) || die "FATAL: can't lock file temp_cost_file.dat\n"; + $tfh->print("${fct}:$lsize:$lcost\n"); + $self->close_export_file($tfh, 1); + } + } + } + } + else + { + $sql_output .= $functions{$fct}{text} . "\n\n"; + } + $self->_restore_comments(\$sql_output); + if ($self->{plsql_pgsql}) { + $sql_output =~ s/(-- REVOKE ALL ON (?:FUNCTION|PROCEDURE) [^;]+ FROM PUBLIC;)/&remove_newline($1)/sge; + } + + my $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; + $sql_header .= "-- Copyright 2000-2020 Gilles DAROLD. All rights reserved.\n"; + $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; + if ($self->{client_encoding}) { + $sql_header .= "SET client_encoding TO '\U$self->{client_encoding}\E';\n\n"; + } + if ($self->{type} ne 'TABLE') { + $sql_header .= $self->set_search_path(); + } + $sql_header .= "\\set ON_ERROR_STOP ON\n\n" if ($self->{stop_on_error}); + $sql_header .= "SET check_function_bodies = false;\n\n" if (!$self->{function_check}); + $sql_header = '' if ($self->{no_header}); + + if ($self->{file_per_function}) { + $self->dump($sql_header . $sql_output, $fhdl); + $self->close_export_file($fhdl); + $sql_output = ''; + } + } + + my $t1 = Benchmark->new; + my $td = timediff($t1, $t0); + $self->logit("Translating of $fct_count functions took: " . timestr($td) . "\n", 1); + + return ($sql_output, $lsize, $lcost); +} + +sub _replace_declare_var +{ + my ($self, $code) = @_; + + if ($$code =~ s/\b(DECLARE\s+(?:.*?)\s+BEGIN)/\%DECLARE\%/is) { + my $declare = $1; + # Collect user defined function + while ($declare =~ s/\b([^\s]+)\s+EXCEPTION\s*;//i) { + my $e = lc($1); + if (!exists $Ora2Pg::PLSQL::EXCEPTION_MAP{"\U$e\L"} && !grep(/^$e$/, values %Ora2Pg::PLSQL::EXCEPTION_MAP) && !exists $self->{custom_exception}{$e}) { + $self->{custom_exception}{$e} = $self->{exception_id}++; + } + } + $declare =~ s/PRAGMA\s+EXCEPTION_INIT[^;]*;//igs; + if ($self->{is_mysql}) { + ($$code, $declare) = Ora2Pg::MySQL::replace_mysql_variables($self, $$code, $declare); + } + $$code =~ s/\%DECLARE\%/$declare/is; + } elsif ($self->{is_mysql}) { + ($$code, $declare) = Ora2Pg::MySQL::replace_mysql_variables($self, $$code, $declare); + $$code = "DECLARE\n" . $declare . "\n" . $$code if ($declare); + } + + # Replace call to raise exception + foreach my $e (keys %{$self->{custom_exception}}) { + $$code =~ s/\bRAISE\s+$e\b/RAISE EXCEPTION '$e' USING ERRCODE = '$self->{custom_exception}{$e}'/igs; + $$code =~ s/(\s+WHEN\s+)$e\s+/$1SQLSTATE '$self->{custom_exception}{$e}' /igs; + } + +} + +# Routine used to save the file to update in pass2 of translation +sub save_filetoupdate_list +{ + my ($self, $pname, $ftcname, $file_name) = @_; + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + + my $tfh = $self->append_export_file($dirprefix . 'temp_pass2_file.dat', 1); + flock($tfh, 2) || die "FATAL: can't lock file temp_pass2_file.dat\n"; + $tfh->print("${pname}:${ftcname}:$file_name\n"); + $self->close_export_file($tfh, 1); +} + +=head2 _set_file_header + +Returns a string containing the common header of each output file. + +=cut + +sub _set_file_header +{ + my $self = shift(); + + return '' if ($self->{no_header}); + + my $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; + $sql_header .= "-- Copyright 2000-2020 Gilles DAROLD. All rights reserved.\n"; + $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; + if ($self->{client_encoding}) + { + $sql_header .= "SET client_encoding TO '\U$self->{client_encoding}\E';\n\n"; + } + if ($self->{type} ne 'TABLE') + { + $sql_header .= $self->set_search_path(); + } + $sql_header .= "\\set ON_ERROR_STOP ON\n\n" if ($self->{stop_on_error}); + $sql_header .= "SET check_function_bodies = false;\n\n" if (!$self->{function_check}); + + return $sql_header; +} + +=head2 export_view + +Export Oracle view into PostgreSQL compatible SQL statements. + +=cut + +sub export_view +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add views definition...\n", 1); + + # Read DML from file if any + if ($self->{input_file}) { + $self->read_view_from_file(); + } + my $nothing = 0; + $self->dump($sql_header); + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + my $i = 1; + my $num_total_view = scalar keys %{$self->{views}}; + %ordered_views = %{$self->{views}}; + my $count_view = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_view); + foreach my $view (sort sort_view_by_iter keys %ordered_views) + { + $self->logit("\tAdding view $view...\n", 1); + if (!$self->{quiet} && !$self->{debug} && ($count_view % $PGBAR_REFRESH) == 0) + { + print STDERR $self->progress_bar($i, $num_total_view, 25, '=', 'views', "generating $view" ), "\r"; + } + $count_view++; + my $fhdl = undef; + if ($self->{file_per_table}) + { + my $file_name = "$dirprefix${view}_$self->{output}"; + $file_name =~ s/\.(gz|bz2)$//; + $self->dump("\\i$self->{psql_relative_path} $file_name\n"); + $self->logit("Dumping to one file per view : ${view}_$self->{output}\n", 1); + $fhdl = $self->open_export_file("${view}_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), $file_name); + } else { + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), "$dirprefix$self->{output}"); + } + $self->_remove_comments(\$self->{views}{$view}{text}); + if (!$self->{pg_supports_checkoption}) { + $self->{views}{$view}{text} =~ s/\s*WITH\s+CHECK\s+OPTION//is; + } + # Remove unsupported definitions from the ddl statement + $self->{views}{$view}{text} =~ s/\s*WITH\s+READ\s+ONLY//is; + $self->{views}{$view}{text} =~ s/\s*OF\s+([^\s]+)\s+(WITH|UNDER)\s+[^\)]+\)//is; + $self->{views}{$view}{text} =~ s/\s*OF\s+XMLTYPE\s+[^\)]+\)//is; + $self->{views}{$view}{text} = $self->_format_view($view, $self->{views}{$view}{text}); + my $tmpv = $view; + if (exists $self->{replaced_tables}{"\L$tmpv\E"} && $self->{replaced_tables}{"\L$tmpv\E"}) + { + $self->logit("\tReplacing table $tmpv as " . $self->{replaced_tables}{lc($tmpv)} . "...\n", 1); + $tmpv = $self->{replaced_tables}{lc($tmpv)}; + } + if ($self->{export_schema} && !$self->{schema} && ($tmpv =~ /^([^\.]+)\./) ) { + $sql_output .= $self->set_search_path($1) . "\n"; + } + $tmpv = $self->quote_object_name($tmpv); + + if (!@{$self->{views}{$view}{alias}}) + { + $sql_output .= "CREATE$self->{create_or_replace} VIEW $tmpv AS "; + $sql_output .= $self->{views}{$view}{text}; + $sql_output .= ';' if ($sql_output !~ /;\s*$/s); + $sql_output .= "\n"; + if ($self->{estimate_cost}) { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $self->{views}{$view}{text}, 'VIEW'); + $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'VIEW'}; + $cost_value += $cost; + $sql_output .= "\n-- Estimed cost of view [ $view ]: " . sprintf("%2.2f", $cost); + } + $sql_output .= "\n"; + } + else + { + $sql_output .= "CREATE$self->{create_or_replace} VIEW $tmpv ("; + my $count = 0; + my %col_to_replace = (); + foreach my $d (@{$self->{views}{$view}{alias}}) + { + if ($count == 0) { + $count = 1; + } else { + $sql_output .= ", "; + } + # Change column names + my $fname = $d->[0]; + if (exists $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"}) + { + $self->logit("\tReplacing column \L$d->[0]\E as " . $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"} . "...\n", 1); + $fname = $self->{replaced_cols}{"\L$view\E"}{"\L$fname\E"}; + } + $sql_output .= $self->quote_object_name($fname); + } + $sql_output .= ") AS " . $self->{views}{$view}{text}; + $sql_output .= ';' if ($sql_output !~ /;\s*$/s); + $sql_output .= "\n"; + if ($self->{estimate_cost}) + { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $self->{views}{$view}{text}, 'VIEW'); + $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'VIEW'}; + $cost_value += $cost; + $sql_output .= "\n-- Estimed cost of view [ $view ]: " . sprintf("%2.2f", $cost); + } + $sql_output .= "\n"; + } + + if ($self->{force_owner}) + { + my $owner = $self->{views}{$view}{owner}; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER VIEW $tmpv OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + + # Add comments on view and columns + if (!$self->{disable_comment}) + { + if ($self->{views}{$view}{comment}) + { + $sql_output .= "COMMENT ON VIEW $tmpv "; + $self->{views}{$view}{comment} =~ s/'/''/gs; + $sql_output .= " IS E'" . $self->{views}{$view}{comment} . "';\n\n"; + } + + foreach my $f (sort { lc{$a} cmp lc($b) } keys %{$self->{views}{$view}{column_comments}}) + { + next unless $self->{views}{$view}{column_comments}{$f}; + $self->{views}{$view}{column_comments}{$f} =~ s/'/''/gs; + # Change column names + my $fname = $f; + if (exists $self->{replaced_cols}{"\L$view\E"}{"\L$f\E"} && $self->{replaced_cols}{"\L$view\E"}{"\L$f\E"}) { + $fname = $self->{replaced_cols}{"\L$view\E"}{"\L$f\E"}; + } + $sql_output .= "COMMENT ON COLUMN " . $self->quote_object_name("$tmpv.$fname") + . " IS E'" . $self->{views}{$view}{column_comments}{$f} . "';\n"; + } + } + + if ($self->{file_per_table}) + { + $self->dump($sql_header . $sql_output, $fhdl); + $self->_restore_comments(\$sql_output); + $self->close_export_file($fhdl); + $sql_output = ''; + } + $nothing++; + $i++; + + } + %ordered_views = (); + + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_view, 25, '=', 'views', 'end of output.'), "\n"; + } + + if (!$nothing) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } else { + $sql_output .= "\n"; + } + + $self->dump($sql_output); + + return; +} + +=head2 export_mview + +Export Oracle materialized view into PostgreSQL compatible SQL statements. + +=cut + +sub export_mview +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add materialized views definition...\n", 1); + + my $nothing = 0; + $self->dump($sql_header) if ($self->{file_per_table} && !$self->{pg_dsn}); + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + if ($self->{plsql_pgsql} && !$self->{pg_supports_mview}) { + my $sqlout = qq{ +$sql_header + +CREATE TABLE materialized_views ( +mview_name text NOT NULL PRIMARY KEY, +view_name text NOT NULL, +iname text, +last_refresh TIMESTAMP WITH TIME ZONE +); + +CREATE OR REPLACE FUNCTION create_materialized_view(text, text, text) +RETURNS VOID +AS \$\$ +DECLARE +mview ALIAS FOR \$1; -- name of the materialized view to create +vname ALIAS FOR \$2; -- name of the related view +iname ALIAS FOR \$3; -- name of the colum of mview to used as unique key +entry materialized_views%ROWTYPE; +BEGIN +EXECUTE 'SELECT * FROM materialized_views WHERE mview_name = ' || quote_literal(mview) || '' INTO entry; +IF entry.iname IS NOT NULL THEN +RAISE EXCEPTION 'Materialized view % already exist.', mview; +END IF; + +EXECUTE 'REVOKE ALL ON ' || quote_ident(vname) || ' FROM PUBLIC'; +EXECUTE 'GRANT SELECT ON ' || quote_ident(vname) || ' TO PUBLIC'; +EXECUTE 'CREATE TABLE ' || quote_ident(mview) || ' AS SELECT * FROM ' || quote_ident(vname); +EXECUTE 'REVOKE ALL ON ' || quote_ident(mview) || ' FROM PUBLIC'; +EXECUTE 'GRANT SELECT ON ' || quote_ident(mview) || ' TO PUBLIC'; +INSERT INTO materialized_views (mview_name, view_name, iname, last_refresh) +VALUES ( +quote_literal(mview), +quote_literal(vname), +quote_literal(iname), +CURRENT_TIMESTAMP +); +IF iname IS NOT NULL THEN +EXECUTE 'CREATE INDEX ' || quote_ident(mview) || '_' || quote_ident(iname) || '_idx ON ' || quote_ident(mview) || '(' || quote_ident(iname) || ')'; +END IF; + +RETURN; +END +\$\$ +SECURITY DEFINER +LANGUAGE plpgsql; + +CREATE OR REPLACE FUNCTION drop_materialized_view(text) RETURNS VOID +AS +\$\$ +DECLARE +mview ALIAS FOR \$1; +entry materialized_views%ROWTYPE; +BEGIN +EXECUTE 'SELECT * FROM materialized_views WHERE mview_name = ''' || quote_literal(mview) || '''' INTO entry; +IF entry.iname IS NULL THEN +RAISE EXCEPTION 'Materialized view % does not exist.', mview; +END IF; + +IF entry.iname IS NOT NULL THEN +EXECUTE 'DROP INDEX ' || quote_ident(mview) || '_' || entry.iname || '_idx'; +END IF; +EXECUTE 'DROP TABLE ' || quote_ident(mview); +EXECUTE 'DELETE FROM materialized_views WHERE mview_name=''' || quote_literal(mview) || ''''; + +RETURN; +END +\$\$ +SECURITY DEFINER +LANGUAGE plpgsql ; + +CREATE OR REPLACE FUNCTION refresh_full_materialized_view(text) RETURNS VOID +AS \$\$ +DECLARE +mview ALIAS FOR \$1; +entry materialized_views%ROWTYPE; +BEGIN +EXECUTE 'SELECT * FROM materialized_views WHERE mview_name = ''' || quote_literal(mview) || '''' INTO entry; +IF entry.iname IS NULL THEN +RAISE EXCEPTION 'Materialized view % does not exist.', mview; +END IF; + +IF entry.iname IS NOT NULL THEN +EXECUTE 'DROP INDEX ' || quote_ident(mview) || '_' || entry.iname || '_idx'; +END IF; +EXECUTE 'TRUNCATE ' || quote_ident(mview); +EXECUTE 'INSERT INTO ' || quote_ident(mview) || ' SELECT * FROM ' || entry.view_name; +EXECUTE 'UPDATE materialized_views SET last_refresh=CURRENT_TIMESTAMP WHERE mview_name=''' || quote_literal(mview) || ''''; + +IF entry.iname IS NOT NULL THEN +EXECUTE 'CREATE INDEX ' || quote_ident(mview) || '_' || entry.iname || '_idx ON ' || quote_ident(mview) || '(' || entry.iname || ')'; +END IF; + +RETURN; +END +\$\$ +SECURITY DEFINER +LANGUAGE plpgsql ; + +}; + $self->dump($sqlout); + } + my $i = 1; + my $num_total_mview = scalar keys %{$self->{materialized_views}}; + my $count_mview = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_mview); + foreach my $view (sort { $a cmp $b } keys %{$self->{materialized_views}}) + { + $self->logit("\tAdding materialized view $view...\n", 1); + if (!$self->{quiet} && !$self->{debug} && ($count_mview % $PGBAR_REFRESH) == 0) { + print STDERR $self->progress_bar($i, $num_total_mview, 25, '=', 'materialized views', "generating $view" ), "\r"; + } + $count_mview++; + my $fhdl = undef; + if ($self->{file_per_table} && !$self->{pg_dsn}) { + my $file_name = "$dirprefix${view}_$self->{output}"; + $file_name =~ s/\.(gz|bz2)$//; + $self->dump("\\i$self->{psql_relative_path} $file_name\n"); + $self->logit("Dumping to one file per materialized view : ${view}_$self->{output}\n", 1); + $fhdl = $self->open_export_file("${view}_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), $file_name); + } else { + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($view), "$dirprefix$self->{output}"); + } + if (!$self->{plsql_pgsql}) { + $sql_output .= "CREATE MATERIALIZED VIEW $view\n"; + $sql_output .= "BUILD $self->{materialized_views}{$view}{build_mode}\n"; + $sql_output .= "REFRESH $self->{materialized_views}{$view}{refresh_method} ON $self->{materialized_views}{$view}{refresh_mode}\n"; + $sql_output .= "ENABLE QUERY REWRITE" if ($self->{materialized_views}{$view}{rewritable}); + $sql_output .= "AS $self->{materialized_views}{$view}{text}"; + $sql_output .= " USING INDEX" if ($self->{materialized_views}{$view}{no_index}); + $sql_output .= " USING NO INDEX" if (!$self->{materialized_views}{$view}{no_index}); + $sql_output .= ";\n\n"; + + # Set the index definition + my ($idx, $fts_idx) = $self->_create_indexes($view, 0, %{$self->{materialized_views}{$view}{indexes}}); + $sql_output .= "$idx$fts_idx\n\n"; + } else { + $self->{materialized_views}{$view}{text} = $self->_format_view($view, $self->{materialized_views}{$view}{text}); + if (!$self->{preserve_case}) { + $self->{materialized_views}{$view}{text} =~ s/"//gs; + } + if ($self->{export_schema} && !$self->{schema} && ($view =~ /^([^\.]+)\./) ) { + $sql_output .= $self->set_search_path($1) . "\n"; + } + $self->{materialized_views}{$view}{text} =~ s/^PERFORM/SELECT/; + if (!$self->{pg_supports_mview}) { + $sql_output .= "CREATE VIEW \L$view\E_mview AS\n"; + $sql_output .= $self->{materialized_views}{$view}{text}; + $sql_output .= ";\n\n"; + $sql_output .= "SELECT create_materialized_view('\L$view\E','\L$view\E_mview', change with the name of the colum to used for the index);\n\n\n"; + + if ($self->{force_owner}) { + my $owner = $self->{materialized_views}{$view}{owner}; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER VIEW " . $self->quote_object_name($view . '_mview') + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + } else { + $sql_output .= "CREATE MATERIALIZED VIEW \L$view\E AS\n"; + $sql_output .= $self->{materialized_views}{$view}{text}; + if ($self->{materialized_views}{$view}{build_mode} eq 'DEFERRED') { + $sql_output .= " WITH NO DATA"; + } + $sql_output .= ";\n"; + # Set the index definition + my ($idx, $fts_idx) = $self->_create_indexes($view, 0, %{$self->{materialized_views}{$view}{indexes}}); + $sql_output .= "$idx$fts_idx\n\n"; + } + } + if ($self->{force_owner}) { + my $owner = $self->{materialized_views}{$view}{owner}; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER MATERIALIZED VIEW " . $self->quote_object_name($view) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + + if ($self->{file_per_table} && !$self->{pg_dsn}) { + $self->dump($sql_header . $sql_output, $fhdl); + $self->close_export_file($fhdl); + $sql_output = ''; + } + $nothing++; + $i++; + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_mview, 25, '=', 'materialized views', 'end of output.'), "\n"; + } + if (!$nothing) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_output); + + return; +} + +=head2 export_grant + +Export Oracle user grants into PostgreSQL compatible SQL statements. + +=cut + +sub export_grant +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add users/roles/grants privileges...\n", 1); + + my $grants = ''; + my $users = ''; + + # Read DML from file if any + if ($self->{input_file}) { + $self->read_grant_from_file(); + } + + # Do not create privilege defintiion if object type is USER + delete $self->{grants} if ($self->{grant_object} && $self->{grant_object} eq 'USER'); + + # Add privilege definition + foreach my $table (sort {"$self->{grants}{$a}{type}.$a" cmp "$self->{grants}{$b}{type}.$b" } keys %{$self->{grants}}) { + my $realtable = lc($table); + my $obj = $self->{grants}{$table}{type} || 'TABLE'; + if ($self->{export_schema} && $self->{schema}) { + $realtable = $self->quote_object_name("$self->{schema}.$table"); + } elsif ($self->{preserve_case}) { + $realtable = $self->quote_object_name($table); + } + $grants .= "-- Set priviledge on $self->{grants}{$table}{type} $table\n"; + + my $ownee = $self->quote_object_name($self->{grants}{$table}{owner}); + + my $wgrantoption = ''; + if ($self->{grants}{$table}{grantable}) { + $wgrantoption = ' WITH GRANT OPTION'; + } + if ($self->{grants}{$table}{type} ne 'PACKAGE BODY') { + if ($self->{grants}{$table}{owner}) { + if (grep(/^$self->{grants}{$table}{owner}$/, @{$self->{roles}{roles}})) { + $grants .= "ALTER $obj $realtable OWNER TO ROLE $ownee;\n"; + $obj = '' if (!grep(/^$obj$/, 'FUNCTION', 'SEQUENCE','SCHEMA','TABLESPACE')); + $grants .= "GRANT ALL ON $obj $realtable TO ROLE $ownee$wgrantoption;\n"; + } else { + $grants .= "ALTER $obj $realtable OWNER TO $ownee;\n"; + $obj = '' if (!grep(/^$obj$/, 'FUNCTION', 'SEQUENCE','SCHEMA','TABLESPACE')); + $grants .= "GRANT ALL ON $obj $realtable TO $ownee$wgrantoption;\n"; + } + } + if (grep(/^$self->{grants}{$table}{type}$/, 'FUNCTION', 'SEQUENCE','SCHEMA','TABLESPACE')) { + $grants .= "REVOKE ALL ON $self->{grants}{$table}{type} $realtable FROM PUBLIC;\n"; + } else { + $grants .= "REVOKE ALL ON $realtable FROM PUBLIC;\n"; + } + } else { + if ($self->{grants}{$table}{owner}) { + if (grep(/^$self->{grants}{$table}{owner}$/, @{$self->{roles}{roles}})) { + $grants .= "ALTER SCHEMA $realtable OWNER TO ROLE $ownee;\n"; + $grants .= "GRANT ALL ON SCHEMA $realtable TO ROLE $ownee$wgrantoption;\n"; + } else { + $grants .= "ALTER SCHEMA $realtable OWNER TO $ownee;\n"; + $grants .= "GRANT ALL ON SCHEMA $realtable TO $ownee$wgrantoption;\n"; + } + } + $grants .= "REVOKE ALL ON SCHEMA $realtable FROM PUBLIC;\n"; + } + foreach my $usr (sort keys %{$self->{grants}{$table}{privilege}}) { + my $agrants = ''; + foreach my $g (@GRANTS) { + $agrants .= "$g," if (grep(/^$g$/i, @{$self->{grants}{$table}{privilege}{$usr}})); + } + $agrants =~ s/,$//; + $usr = $self->quote_object_name($usr); + if ($self->{grants}{$table}{type} ne 'PACKAGE BODY') { + if (grep(/^$self->{grants}{$table}{type}$/, 'FUNCTION', 'SEQUENCE','SCHEMA','TABLESPACE', 'TYPE')) { + $grants .= "GRANT $agrants ON $obj $realtable TO $usr$wgrantoption;\n"; + } else { + $grants .= "GRANT $agrants ON $realtable TO $usr$wgrantoption;\n"; + } + } else { + $grants .= "GRANT USAGE ON SCHEMA $realtable TO $usr$wgrantoption;\n"; + $grants .= "GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA $realtable TO $usr$wgrantoption;\n"; + } + } + $grants .= "\n"; + } + + # Do not create user when privilege on an object type is asked + delete $self->{roles} if ($self->{grant_object} && $self->{grant_object} ne 'USER'); + + foreach my $r (@{$self->{roles}{owner}}, @{$self->{roles}{grantee}}) + { + my $secret = 'change_my_secret'; + if ($self->{gen_user_pwd}) { + $secret = &randpattern("CccnCccn"); + } + $sql_header .= "CREATE " . ($self->{roles}{type}{$r} ||'USER') . " $r"; + $sql_header .= " WITH PASSWORD '$secret'" if ($self->{roles}{password_required}{$r} ne 'NO'); + # It's difficult to parse all oracle privilege. So if one admin option is set we set all PG admin option. + if (grep(/YES|1/, @{$self->{roles}{$r}{admin_option}})) { + $sql_header .= " CREATEDB CREATEROLE CREATEUSER INHERIT"; + } + if ($self->{roles}{type}{$r} eq 'USER') { + $sql_header .= " LOGIN"; + } + if (exists $self->{roles}{role}{$r}) { + $users .= " IN ROLE " . join(',', @{$self->{roles}{role}{$r}}); + } + $sql_header .= ";\n"; + } + if (!$grants) { + $grants = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $sql_output .= "\n" . $grants . "\n" if ($grants); + + $self->_restore_comments(\$grants); + $self->dump($sql_header . $sql_output); + + return; +} + +=head2 export_sequence + +Export Oracle sequence into PostgreSQL compatible SQL statements. + +=cut + +sub export_sequence +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add sequences definition...\n", 1); + + # Read DML from file if any + if ($self->{input_file}) { + $self->read_sequence_from_file(); + } + my $i = 1; + my $num_total_sequence = $#{$self->{sequences}} + 1; + my $count_seq = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_sequence); + if ($self->{export_schema}) { + $sql_output .= "CREATE SCHEMA IF NOT EXISTS " . $self->quote_object_name($self->{pg_schema} || $self->{schema}) . ";\n"; + } + foreach my $seq (sort { $a->[0] cmp $b->[0] } @{$self->{sequences}}) + { + if (!$self->{quiet} && !$self->{debug} && ($count_seq % $PGBAR_REFRESH) == 0) { + print STDERR $self->progress_bar($i, $num_total_sequence, 25, '=', 'sequences', "generating $seq->[0]" ), "\r"; + } + $count_seq++; + my $cache = ''; + $cache = $seq->[5] if ($seq->[5]); + my $cycle = ''; + $cycle = ' CYCLE' if ($seq->[6] eq 'Y'); + if ($self->{export_schema} && !$self->{schema}) { + $seq->[0] = $seq->[7] . '.' . $seq->[0]; + } + $sql_output .= "CREATE SEQUENCE " . $self->quote_object_name($seq->[0]) . " INCREMENT $seq->[3]"; + if ($seq->[1] eq '' || $seq->[1] < (-2**63-1)) { + $sql_output .= " NO MINVALUE"; + } else { + $sql_output .= " MINVALUE $seq->[1]"; + } + # Max value lower than start value are not allowed + if (($seq->[2] > 0) && ($seq->[2] < $seq->[4])) { + $seq->[2] = $seq->[4]; + } + if ($seq->[2] eq '' || $seq->[2] > (2**63-1)) { + $sql_output .= " NO MAXVALUE"; + } else { + $seq->[2] = 9223372036854775807 if ($seq->[2] > 9223372036854775807); + $sql_output .= " MAXVALUE $seq->[2]"; + } + $sql_output .= " START $seq->[4]"; + $sql_output .= " CACHE $cache" if ($cache ne ''); + $sql_output .= "$cycle;\n"; + + if ($self->{force_owner}) { + my $owner = $seq->[7]; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER SEQUENCE " . $self->quote_object_name($seq->[0]) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + $i++; + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_sequence, 25, '=', 'sequences', 'end of output.'), "\n"; + } + if (!$sql_output) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_header . $sql_output); + + return; +} + +=head2 export_dblink + +Export Oracle sequence into PostgreSQL compatible SQL statements. + +=cut + +sub export_dblink +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add dblink definition...\n", 1); + + # Read DML from file if any + if ($self->{input_file}) { + $self->read_dblink_from_file(); + } + my $i = 1; + my $num_total_dblink = scalar keys %{$self->{dblink}}; + + foreach my $db (sort { $a cmp $b } keys %{$self->{dblink}}) { + + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i, $num_total_dblink, 25, '=', 'dblink', "generating $db" ), "\r"; + } + $sql_output .= "CREATE SERVER " . $self->quote_object_name($db); + if (!$self->{is_mysql}) { + $sql_output .= " FOREIGN DATA WRAPPER oracle_fdw OPTIONS (dbserver '$self->{dblink}{$db}{host}');\n"; + } else { + $sql_output .= " FOREIGN DATA WRAPPER mysql_fdw OPTIONS (host '$self->{dblink}{$db}{host}'"; + $sql_output .= ", port '$self->{dblink}{$db}{port}'" if ($self->{dblink}{$db}{port}); + $sql_output .= ");\n"; + } + if ($self->{dblink}{$db}{password} ne 'NONE') { + $self->{dblink}{$db}{password} ||= 'secret'; + $self->{dblink}{$db}{password} = ", password '$self->{dblink}{$db}{password}'"; + } + if ($self->{dblink}{$db}{username}) { + $sql_output .= "CREATE USER MAPPING FOR " . $self->quote_object_name($self->{dblink}{$db}{username}) + . " SERVER " . $self->quote_object_name($db) + . " OPTIONS (user '" . $self->quote_object_name($self->{dblink}{$db}{user}) + . "' $self->{dblink}{$db}{password});\n"; + } + + if ($self->{force_owner}) { + my $owner = $self->{dblink}{$db}{owner}; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER FOREIGN DATA WRAPPER " . $self->quote_object_name($db) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + $i++; + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_dblink, 25, '=', 'dblink', 'end of output.'), "\n"; + } + if (!$sql_output) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_header . $sql_output); + + return; +} + +=head2 export_directory + +Export Oracle directory into PostgreSQL compatible SQL statements. + +=cut + +sub export_directory +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add directory definition...\n", 1); + + # Read DML from file if any + if ($self->{input_file}) { + $self->read_directory_from_file(); + } + my $i = 1; + my $num_total_directory = scalar keys %{$self->{directory}}; + + foreach my $db (sort { $a cmp $b } keys %{$self->{directory}}) { + + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i, $num_total_directory, 25, '=', 'directory', "generating $db" ), "\r"; + } + $sql_output .= "INSERT INTO external_file.directories (directory_name,directory_path) VALUES ('$db', '$self->{directory}{$db}{path}');\n"; + foreach my $owner (keys %{$self->{directory}{$db}{grantee}}) { + my $write = 'false'; + $write = 'true' if ($self->{directory}{$db}{grantee}{$owner} =~ /write/i); + $sql_output .= "INSERT INTO external_file.directory_roles(directory_name,directory_role,directory_read,directory_write) VALUES ('$db','" . $self->quote_object_name($owner) . "', true, $write);\n"; + } + $i++; + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_directory, 25, '=', 'directory', 'end of output.'), "\n"; + } + if (!$sql_output) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_header . $sql_output); + + return; +} + +=head2 export_trigger + +Export Oracle trigger into PostgreSQL compatible SQL statements. + +=cut + +sub export_trigger +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add triggers definition...\n", 1); + + $self->dump($sql_header); + # Read DML from file if any + if ($self->{input_file}) { + $self->read_trigger_from_file(); + } + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + my $nothing = 0; + my $i = 1; + my $num_total_trigger = $#{$self->{triggers}} + 1; + my $count_trig = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_trigger); + foreach my $trig (sort {$a->[0] cmp $b->[0]} @{$self->{triggers}}) + { + if (!$self->{quiet} && !$self->{debug} && ($count_trig % $PGBAR_REFRESH) == 0) { + print STDERR $self->progress_bar($i, $num_total_trigger, 25, '=', 'triggers', "generating $trig->[0]" ), "\r"; + } + $count_trig++; + my $fhdl = undef; + if ($self->{file_per_function}) + { + my $f = "$dirprefix$trig->[0]_$self->{output}"; + $f =~ s/\.(?:gz|bz2)$//i; + $self->dump("\\i$self->{psql_relative_path} $f\n"); + $self->logit("Dumping to one file per trigger : $trig->[0]_$self->{output}\n", 1); + $fhdl = $self->open_export_file("$trig->[0]_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($trig->[0]), "$dirprefix$trig->[0]_$self->{output}"); + } + else + { + $self->save_filetoupdate_list("ORA2PG_$self->{type}", lc($trig->[0]), "$dirprefix$self->{output}"); + } + $trig->[1] =~ s/\s*EACH ROW//is; + chomp($trig->[4]); + + $trig->[4] =~ s/([^\*])[;\/]$/$1/; + + $self->logit("\tDumping trigger $trig->[0] defined on table $trig->[3]...\n", 1); + my $tbname = $self->get_replaced_tbname($trig->[3]); + + # Store current trigger table name for possible use in outer join translation + $self->{current_trigger_table} = $trig->[3]; + + # Replace column name in function code + if (exists $self->{replaced_cols}{"\L$trig->[3]\E"}) + { + foreach my $coln (sort keys %{$self->{replaced_cols}{"\L$trig->[3]\E"}}) + { + $self->logit("\tReplacing column \L$coln\E as " . $self->{replaced_cols}{"\L$trig->[3]\E"}{"\L$coln\E"} . "...\n", 1); + my $cname = $self->{replaced_cols}{"\L$trig->[3]\E"}{"\L$coln\E"}; + $cname = $self->quote_object_name($cname); + $trig->[4] =~ s/(OLD|NEW)\.$coln\b/$1\.$cname/igs; + $trig->[6] =~ s/\b$coln\b/$self->{replaced_cols}{"\L$trig->[3]\E"}{"\L$coln\E"}/is; + } + } + # Extract columns specified in the UPDATE OF ... ON clause + my $cols = ''; + if ($trig->[2] =~ /UPDATE/ && $trig->[6] =~ /UPDATE\s+OF\s+(.*?)\s+ON/i) + { + my @defs = split(/\s*,\s*/, $1); + $cols = ' OF '; + foreach my $c (@defs) { + $cols .= $self->quote_object_name($c) . ','; + } + $cols =~ s/,$//; + } + + if ($self->{export_schema} && !$self->{schema}) { + $sql_output .= $self->set_search_path($trig->[8]) . "\n"; + } + # Check if it's like a pg rule + $self->_remove_comments(\$trig->[4]); + if (!$self->{pg_supports_insteadof} && $trig->[1] =~ /INSTEAD OF/) + { + if ($self->{plsql_pgsql}) + { + $trig->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[4]); + $self->_replace_declare_var(\$trig->[4]); + } + $sql_output .= "CREATE$self->{create_or_replace} RULE " . $self->quote_object_name($trig->[0]) + . " AS\n\tON " . $self->quote_object_name($trig->[2]) + . " TO " . $self->quote_object_name($tbname) + . "\n\tDO INSTEAD\n(\n\t$trig->[4]\n);\n\n"; + if ($self->{force_owner}) + { + my $owner = $trig->[8]; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER RULE " . $self->quote_object_name($trig->[0]) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + } + else + { + # Replace direct call of a stored procedure in triggers + if ($trig->[7] eq 'CALL') + { + if ($self->{plsql_pgsql}) + { + $trig->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[4]); + $self->_replace_declare_var(\$trig->[4]); + } + $trig->[4] = "BEGIN\nPERFORM $trig->[4];\nEND;"; + } + else + { + my $ret_kind = 'RETURN NEW;'; + if (uc($trig->[2]) eq 'DELETE') { + $ret_kind = 'RETURN OLD;'; + } elsif (uc($trig->[2]) =~ /DELETE/) { + $ret_kind = "IF TG_OP = 'DELETE' THEN\n\tRETURN OLD;\nELSE\n\tRETURN NEW;\nEND IF;\n"; + } + if ($self->{plsql_pgsql}) + { + # Add a semi colon if none + if ($trig->[4] !~ /\bBEGIN\b/i) + { + chomp($trig->[4]); + $trig->[4] .= ';' if ($trig->[4] !~ /;\s*$/s); + $trig->[4] = "BEGIN\n$trig->[4]\n$ret_kind\nEND;"; + } + $trig->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[4]); + $self->_replace_declare_var(\$trig->[4]); + + # When an exception statement is used enclosed everything + # in a block before returning NEW + if ($trig->[4] =~ /EXCEPTION(.*?)\b(END[;]*)[\s\/]*$/is) + { + $trig->[4] =~ s/^\s*BEGIN/BEGIN\n BEGIN/ism; + $trig->[4] =~ s/\b(END[;]*)[\s\/]*$/ END;\n$1/is; + } + # Add return statement. + $trig->[4] =~ s/\b(END[;]*)(\s*\%ORA2PG_COMMENT\d+\%\s*)?[\s\/]*$/$ret_kind\n$1$2/igs; + # Look at function header to convert sql type + my @parts = split(/BEGIN/i, $trig->[4]); + if ($#parts > 0) + { + if (!$self->{is_mysql}) { + $parts[0] = Ora2Pg::PLSQL::replace_sql_type($parts[0], $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + } else { + $parts[0] = Ora2Pg::MySQL::replace_sql_type($parts[0], $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + } + } + $trig->[4] = join('BEGIN', @parts); + $trig->[4] =~ s/\bRETURN\s*;/$ret_kind/igs; + } + } + $sql_output .= "DROP TRIGGER $self->{pg_supports_ifexists} " . $self->quote_object_name($trig->[0]) + . " ON " . $self->quote_object_name($tbname) . " CASCADE;\n"; + my $security = ''; + my $revoke = ''; + my $trig_fctname = $self->quote_object_name("trigger_fct_\L$trig->[0]\E"); + if ($self->{security}{"\U$trig->[0]\E"}{security} eq 'DEFINER') + { + $security = " SECURITY DEFINER"; + $revoke = "-- REVOKE ALL ON FUNCTION $trig_fctname() FROM PUBLIC;\n"; + } + $security = " SECURITY INVOKER" if ($self->{force_security_invoker}); + if ($self->{pg_supports_when} && $trig->[5]) + { + if (!$self->{preserve_case}) + { + $trig->[4] =~ s/"([^"]+)"/\L$1\E/gs; + $trig->[4] =~ s/ALTER TRIGGER\s+[^\s]+\s+ENABLE(;)?//; + } + $sql_output .= "CREATE$self->{create_or_replace} FUNCTION $trig_fctname() RETURNS trigger AS \$BODY\$\n$trig->[4]\n\$BODY\$\n LANGUAGE 'plpgsql'$security;\n$revoke\n"; + if ($self->{force_owner}) + { + my $owner = $trig->[8]; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER FUNCTION $trig_fctname() OWNER TO " . $self->quote_object_name($owner) . ";\n\n"; + } + $self->_remove_comments(\$trig->[6]); + $trig->[6] =~ s/\n+$//s; + $trig->[6] =~ s/^[^\.\s]+\.//; + if (!$self->{preserve_case}) { + $trig->[6] =~ s/"([^"]+)"/\L$1\E/gs; + } + chomp($trig->[6]); + # Remove referencing clause, not supported by PostgreSQL + $trig->[6] =~ s/REFERENCING\s+(.*?)(FOR\s+EACH\s+)/$2/is; + $trig->[6] =~ s/^\s*["]*(?:$trig->[0])["]*//is; + $trig->[6] =~ s/\s+ON\s+([^"\s]+)\s+/" ON " . $self->quote_object_name($1) . " "/ies; + $sql_output .= "CREATE TRIGGER " . $self->quote_object_name($trig->[0]) . "$trig->[6]\n"; + if ($trig->[5]) + { + $self->_remove_comments(\$trig->[5]); + $trig->[5] =~ s/"([^"]+)"/\L$1\E/gs if (!$self->{preserve_case}); + if ($self->{plsql_pgsql}) + { + $trig->[5] = Ora2Pg::PLSQL::convert_plsql_code($self, $trig->[5]); + $self->_replace_declare_var(\$trig->[5]); + } + $sql_output .= "\tWHEN ($trig->[5])\n"; + } + $sql_output .= "\tEXECUTE PROCEDURE $trig_fctname();\n\n"; + } + else + { + if (!$self->{preserve_case}) + { + $trig->[4] =~ s/"([^"]+)"/\L$1\E/gs; + $trig->[4] =~ s/ALTER TRIGGER\s+[^\s]+\s+ENABLE(;)?//; + } + $sql_output .= "CREATE$self->{create_or_replace} FUNCTION $trig_fctname() RETURNS trigger AS \$BODY\$\n$trig->[4]\n\$BODY\$\n LANGUAGE 'plpgsql'$security;\n$revoke\n"; + if ($self->{force_owner}) + { + my $owner = $trig->[8]; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER FUNCTION $trig_fctname() OWNER TO " . $self->quote_object_name($owner) . ";\n\n"; + } + $sql_output .= "CREATE TRIGGER " . $self->quote_object_name($trig->[0]) . "\n\t"; + my $statement = 0; + $statement = 1 if ($trig->[1] =~ s/ STATEMENT//); + $sql_output .= "$trig->[1] $trig->[2]$cols ON " . $self->quote_object_name($tbname) . " "; + if ($statement) { + $sql_output .= "FOR EACH STATEMENT\n"; + } else { + $sql_output .= "FOR EACH ROW\n"; + } + $sql_output .= "\tEXECUTE PROCEDURE $trig_fctname();\n\n"; + } + } + $self->_restore_comments(\$sql_output); + if ($self->{file_per_function}) + { + $self->dump($sql_header . $sql_output, $fhdl); + $self->close_export_file($fhdl); + $sql_output = ''; + } + $nothing++; + $i++; + } + delete $self->{current_trigger_table}; + + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_trigger, 25, '=', 'triggers', 'end of output.'), "\n"; + } + if (!$nothing) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_output); + + return; +} + +=head2 parallelize_statements + +Parallelize SQL statements to import into PostgreSQL. + +=cut + +sub parallelize_statements +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Parse SQL orders to load...\n", 1); + + my $nothing = 0; + #--------------------------------------------------------- + # Load a file containing SQL code to load into PostgreSQL + #--------------------------------------------------------- + my %comments = (); + my @settings = (); + if ($self->{input_file}) + { + $self->{functions} = (); + $self->logit("Reading input SQL orders from file $self->{input_file}...\n", 1); + my $content = $self->read_input_file($self->{input_file}); + # remove comments only, text constants are preserved + $self->_remove_comments(\$content, 1); + $content =~ s/\%ORA2PG_COMMENT\d+\%//gs; + my $query = 1; + foreach my $l (split(/\n/, $content)) + { + chomp($l); + next if ($l =~ /^\s*$/); + # do not parse interactive or session command + next if ($l =~ /^(\\set|\\pset|\\i)/is); + # Put setting change in header to apply them on all parallel session + # This will help to set a special search_path or encoding + if ($l =~ /^SET\s+/i) + { + push(@settings, $l); + next; + } + if ($old_line) + { + $l = $old_line .= ' ' . $l; + $old_line = ''; + } + if ($l =~ /;\s*$/) + { + $self->{queries}{$query} .= "$l\n"; + $query++; + } else { + $self->{queries}{$query} .= "$l\n"; + } + } + } else { + $self->logit("No input file, aborting...\n", 0, 1); + } + + #-------------------------------------------------------- + my $total_queries = scalar keys %{$self->{queries}}; + $self->{child_count} = 0; + foreach my $q (sort {$a <=> $b} keys %{$self->{queries}}) + { + chomp($self->{queries}{$q}); + next if (!$self->{queries}{$q}); + if ($self->{jobs} > 1) + { + while ($self->{child_count} >= $self->{jobs}) + { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) + { + $self->{child_count}--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + spawn sub { + $self->_pload_to_pg($q, $self->{queries}{$q}, @settings); + }; + $self->{child_count}++; + } else { + $self->_pload_to_pg($q, $self->{queries}{$q}, @settings); + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($q, $total_queries, 25, '=', 'queries', "dispatching query #$q" ), "\r"; + } + $nothing++; + } + $self->{queries} = (); + + if (!$total_queries) { + $self->logit("No query to load...\n", 0); + } else { + # Wait for all child end + while ($self->{child_count} > 0) + { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) + { + $self->{child_count}--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR "\n"; + } + } + return; +} + +=head2 translate_query + +Translate Oracle's queries into PostgreSQL compatible statement. + +=cut + +sub translate_query +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Parse queries definition...\n", 1); + $self->dump($sql_header); + + my $nothing = 0; + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + #--------------------------------------------------------- + # Code to use to find queries parser issues, it load a file + # containing the untouched SQL code from Oracle queries + #--------------------------------------------------------- + if ($self->{input_file}) + { + $self->{functions} = (); + $self->logit("Reading input code from file $self->{input_file}...\n", 1); + my $content = $self->read_input_file($self->{input_file}); + $self->_remove_comments(\$content); + my $query = 1; + foreach my $l (split(/(?:^\/$|;\s*$)/m, $content)) + { + chomp($l); + next if ($l =~ /^\s*$/s); + $self->{queries}{$query}{code} = "$l\n"; + $query++; + } + $content = ''; + foreach my $q (keys %{$self->{queries}}) { + $self->_restore_comments(\$self->{queries}{$q}{code}); + } + } + + foreach my $q (sort { $a <=> $b } keys %{$self->{queries}}) + { + if ($self->{queries}{$q}{code} !~ /(SELECT|UPDATE|DELETE|INSERT|DROP|TRUNCATE|CREATE(?:UNIQUE)? INDEX)/is) { + $self->{queries}{$q}{to_be_parsed} = 0; + } else { + $self->{queries}{$q}{to_be_parsed} = 1; + } + } + + #-------------------------------------------------------- + + my $total_size = 0; + my $cost_value = 0; + foreach my $q (sort {$a <=> $b} keys %{$self->{queries}}) + { + $total_size += length($self->{queries}{$q}{code}); + $self->logit("Dumping query $q...\n", 1); + if ($self->{queries}{$q}{to_be_parsed}) { + if ($self->{plsql_pgsql}) { + $self->_remove_comments(\$self->{queries}{$q}{code}); + my $sql_q = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{queries}{$q}{code}); + my $estimate = ''; + if ($self->{estimate_cost}) { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $sql_q, 'QUERY'); + $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'QUERY'}; + $cost_value += $cost; + $estimate = "\n-- Estimed cost of query [ $q ]: " . sprintf("%2.2f", $cost); + } + $self->_restore_comments(\$sql_q); + $sql_output .= $sql_q; + $sql_output .= ';' if ($sql_q !~ /;\s*$/); + $sql_output .= $estimate; + } else { + $sql_output .= $self->{queries}{$q}{code}; + } + } else { + $sql_output .= $self->{queries}{$q}{code}; + $sql_output .= ';' if ($self->{queries}{$q}{code} !~ /;\s*$/); + } + $sql_output .= "\n"; + $nothing++; + } + if ($self->{estimate_cost}) { + $cost_value = sprintf("%2.2f", $cost_value); + my @infos = ( "Total number of queries: ".(scalar keys %{$self->{queries}}).".", + "Total size of queries code: $total_size bytes.", + "Total estimated cost: $cost_value units, ".$self->_get_human_cost($cost_value)."." + ); + $self->logit(join("\n", @infos) . "\n", 1); + map { s/^/-- /; } @infos; + $sql_output .= join("\n", @infos); + } + if (!$nothing) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + $self->dump($sql_output); + + $self->{queries} = (); + + return; +} + +=head2 export_function + +Export Oracle functions into PostgreSQL compatible statement. + +=cut + +sub export_function +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + use constant SQL_DATATYPE => 2; + $self->logit("Add functions definition...\n", 1); + $self->dump($sql_header); + my $nothing = 0; + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + #--------------------------------------------------------- + # Code to use to find function parser issues, it load a file + # containing the untouched PL/SQL code from Oracle Function + #--------------------------------------------------------- + if ($self->{input_file}) + { + $self->{functions} = (); + $self->logit("Reading input code from file $self->{input_file}...\n", 1); + my $content = $self->read_input_file($self->{input_file}); + $self->_remove_comments(\$content); + my @allfct = split(/\n/, $content); + my $fcnm = ''; + my $old_line = ''; + my $language = ''; + foreach my $l (@allfct) { + chomp($l); + $l =~ s/\r//g; + next if ($l =~ /^\s*$/); + if ($old_line) { + $l = $old_line .= ' ' . $l; + $old_line = ''; + } + if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*$/i) { + $old_line = $l; + next; + } + if ($l =~ /^\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*(FUNCTION|PROCEDURE)$/i) { + $old_line = $l; + next; + } + if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*(FUNCTION|PROCEDURE)\s*$/i) { + $old_line = $l; + next; + } + $l =~ s/^\s*CREATE (?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE|DEFINER=[^\s]+)?\s*(FUNCTION|PROCEDURE)/$1/i; + $l =~ s/^\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)/$1/i; + if ($l =~ /^(FUNCTION|PROCEDURE)\s+([^\s\(]+)/i) { + $fcnm = $2; + $fcnm =~ s/"//g; + } + next if (!$fcnm); + if ($l =~ /LANGUAGE\s+([^\s="'><\!\(\)]+)/is) { + $language = $1; + } + $self->{functions}{$fcnm}{text} .= "$l\n"; + + if (!$language) { + if ($l =~ /^END\s+$fcnm(_atx)?\s*;/i) { + $fcnm = ''; + } + } else { + if ($l =~ /;/i) { + $fcnm = ''; + $language = ''; + } + } + } + # Get all metadata from all functions when we are + # reading a file, otherwise it has already been done + foreach my $fct (sort keys %{$self->{functions}}) + { + $self->{functions}{$fct}{text} =~ s/(.*?\b(?:FUNCTION|PROCEDURE)\s+(?:[^\s\(]+))(\s*\%ORA2PG_COMMENT\d+\%\s*)+/$2$1 /is; + my %fct_detail = $self->_lookup_function($self->{functions}{$fct}{text}, ($self->{is_mysql}) ? $fct : undef); + if (!exists $fct_detail{name}) { + delete $self->{functions}{$fct}; + next; + } + $self->{functions}{$fct}{type} = uc($fct_detail{type}); + delete $fct_detail{code}; + delete $fct_detail{before}; + my $sch = 'unknown'; + my $fname = $fct; + if ($fname =~ s/^([^\.\s]+)\.([^\s]+)$/$2/is) { + $sch = $1; + } + $fname =~ s/"//g; + %{$self->{function_metadata}{$sch}{'none'}{$fname}{metadata}} = %fct_detail; + $self->_restore_comments(\$self->{functions}{$fct}{text}); + } + } + + #-------------------------------------------------------- + my $total_size = 0; + my $cost_value = 0; + my $num_total_function = scalar keys %{$self->{functions}}; + my $fct_cost = ''; + my $parallel_fct_count = 0; + unlink($dirprefix . 'temp_cost_file.dat') if ($self->{parallel_tables} > 1 && $self->{estimate_cost}); + + my $t0 = Benchmark->new; + + # Group functions by chunk in multiprocess mode + my $num_chunk = $self->{jobs} || 1; + my @fct_group = (); + my $i = 0; + foreach my $key ( sort keys %{$self->{functions}} ) + { + $fct_group[$i++]{$key} = $self->{functions}{$key}; + $i = 0 if ($i == $num_chunk); + } + my $num_cur_fct = 0; + for ($i = 0; $i <= $#fct_group; $i++) + { + + if ($self->{jobs} > 1) { + $self->logit("Creating a new process to translate functions...\n", 1); + spawn sub { + $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); + }; + $parallel_fct_count++; + } else { + my ($code, $lsize, $lcost) = $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); + $sql_output .= $code; + $total_size += $lsize; + $cost_value += $lcost; + } + $num_cur_fct += scalar keys %{$fct_group[$i]}; + $nothing++; + } + # Wait for all oracle connection terminaison + if ($self->{jobs} > 1) + { + while ($parallel_fct_count) + { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $parallel_fct_count--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + if ($self->{estimate_cost}) { + my $tfh = $self->read_export_file($dirprefix . 'temp_cost_file.dat'); + flock($tfh, 2) || die "FATAL: can't lock file temp_cost_file.dat\n"; + while (my $l = <$tfh>) { + chomp($l); + my ($fname, $fsize, $fcost) = split(/:/, $l); + $total_size += $fsize; + $cost_value += $fcost; + } + $self->close_export_file($tfh, 1); + unlink($dirprefix . 'temp_cost_file.dat'); + } + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($num_cur_fct, $num_total_function, 25, '=', 'functions', 'end of functions export.'), "\n"; + } + if ($self->{estimate_cost}) { + my @infos = ( "Total number of functions: ".(scalar keys %{$self->{functions}}).".", + "Total size of function code: $total_size bytes.", + "Total estimated cost: $cost_value units, ".$self->_get_human_cost($cost_value)."." + ); + $self->logit(join("\n", @infos) . "\n", 1); + map { s/^/-- /; } @infos; + $sql_output .= "\n" . join("\n", @infos); + $sql_output .= "\n" . $fct_cost; + } + if (!$nothing) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_output); + + $self->{functions} = (); + + my $t1 = Benchmark->new; + my $td = timediff($t1, $t0); + $self->logit("Total time to translate all functions with $num_chunk process: " . timestr($td) . "\n", 1); + + return; +} + +=head2 export_procedure + +Export Oracle procedures into PostgreSQL compatible statement. + +=cut + +sub export_procedure +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + use constant SQL_DATATYPE => 2; + $self->logit("Add procedures definition...\n", 1); + my $nothing = 0; + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + $self->dump($sql_header); + #--------------------------------------------------------- + # Code to use to find procedure parser issues, it load a file + # containing the untouched PL/SQL code from Oracle Procedure + #--------------------------------------------------------- + if ($self->{input_file}) + { + $self->{procedures} = (); + $self->logit("Reading input code from file $self->{input_file}...\n", 1); + my $content = $self->read_input_file($self->{input_file}); + $self->_remove_comments(\$content); + my @allfct = split(/\n/, $content); + my $fcnm = ''; + my $old_line = ''; + my $language = ''; + my $first_comment = ''; + foreach my $l (@allfct) + { + $l =~ s/\r//g; + next if ($l =~ /^\/$/); + next if ($l =~ /^\s*$/); + if ($old_line) + { + $l = $old_line .= ' ' . $l; + $old_line = ''; + } + $comment .= $l if ($l =~ /^\%ORA2PG_COMMENT\d+\%$/); + if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*$/i) + { + $old_line = $comment . $l; + $comment = ''; + next; + } + if ($l =~ /^\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)$/i) + { + $old_line = $comment . $l; + $comment = ''; + next; + } + if ($l =~ /^\s*CREATE\s*(?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)\s*$/i) + { + $old_line = $comment . $l; + $comment = ''; + next; + } + $l =~ s/^\s*CREATE (?:OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)/$1/i; + $l =~ s/^\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*(FUNCTION|PROCEDURE)/$1/i; + if ($l =~ /^(FUNCTION|PROCEDURE)\s+([^\s\(]+)/i) + { + $fcnm = $2; + $fcnm =~ s/"//g; + } + next if (!$fcnm); + if ($l =~ /LANGUAGE\s+([^\s="'><\!\(\)]+)/is) { + $language = $1; + } + if ($comment) + { + $self->{procedures}{$fcnm}{text} .= "$comment"; + $comment = ''; + } + $self->{procedures}{$fcnm}{text} .= "$l\n"; + if (!$language) + { + if ($l =~ /^END\s+$fcnm(_atx)?\s*;/i) { + $fcnm = ''; + } + } + else + { + if ($l =~ /;/i) + { + $fcnm = ''; + $language = ''; + } + } + } + + # Get all metadata from all procedures when we are + # reading a file, otherwise it has already been done + foreach my $fct (sort keys %{$self->{procedures}}) + { + $self->{procedures}{$fct}{text} =~ s/(.*?\b(?:FUNCTION|PROCEDURE)\s+(?:[^\s\(]+))(\s*\%ORA2PG_COMMENT\d+\%\s*)+/$2$1 /is; + my %fct_detail = $self->_lookup_function($self->{procedures}{$fct}{text}, ($self->{is_mysql}) ? $fct : undef); + if (!exists $fct_detail{name}) + { + delete $self->{procedures}{$fct}; + next; + } + $self->{procedures}{$fct}{type} = $fct_detail{type}; + delete $fct_detail{code}; + delete $fct_detail{before}; + my $sch = 'unknown'; + my $fname = $fct; + if ($fname =~ s/^([^\.\s]+)\.([^\s]+)$/$2/is) { + $sch = $1; + } + $fname =~ s/"//g; + %{$self->{function_metadata}{$sch}{'none'}{$fname}{metadata}} = %fct_detail; + $self->_restore_comments(\$self->{procedures}{$fct}{text}); + } + } + + #-------------------------------------------------------- + my $total_size = 0; + my $cost_value = 0; + my $num_total_function = scalar keys %{$self->{procedures}}; + my $fct_cost = ''; + my $parallel_fct_count = 0; + unlink($dirprefix . 'temp_cost_file.dat') if ($self->{parallel_tables} > 1 && $self->{estimate_cost}); + + my $t0 = Benchmark->new; + + # Group functions by chunk in multiprocess mode + my $num_chunk = $self->{jobs} || 1; + my @fct_group = (); + my $i = 0; + foreach my $key (sort keys %{$self->{procedures}} ) { + $fct_group[$i++]{$key} = $self->{procedures}{$key}; + $i = 0 if ($i == $num_chunk); + } + my $num_cur_fct = 0; + for ($i = 0; $i <= $#fct_group; $i++) { + if ($self->{jobs} > 1) { + $self->logit("Creating a new process to translate procedures...\n", 1); + spawn sub { + $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); + }; + $parallel_fct_count++; + } else { + my ($code, $lsize, $lcost) = $self->translate_function($num_cur_fct, $num_total_function, %{$fct_group[$i]}); + $sql_output .= $code; + $total_size += $lsize;; + $cost_value += $lcost; + } + $num_cur_fct += scalar keys %{$fct_group[$i]}; + $nothing++; + } + + # Wait for all oracle connection terminaison + if ($self->{jobs} > 1) { + while ($parallel_fct_count) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $parallel_fct_count--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + if ($self->{estimate_cost}) { + my $tfh = $self->read_export_file($dirprefix . 'temp_cost_file.dat'); + flock($tfh, 2) || die "FATAL: can't lock file temp_cost_file.dat\n"; + if (defined $tfh) { + while (my $l = <$tfh>) { + chomp($l); + my ($fname, $fsize, $fcost) = split(/:/, $l); + $total_size += $fsize; + $cost_value += $fcost; + } + $self->close_export_file($tfh, 1); + } + unlink($dirprefix . 'temp_cost_file.dat'); + } + } + if (!$self->{quiet} && !$self->{debug}) + { + print STDERR $self->progress_bar($num_cur_fct, $num_total_function, 25, '=', 'procedures', 'end of procedures export.'), "\n"; + } + if ($self->{estimate_cost}) { + my @infos = ( "Total number of procedures: ".(scalar keys %{$self->{procedures}}).".", + "Total size of procedures code: $total_size bytes.", + "Total estimated cost: $cost_value units, ".$self->_get_human_cost($cost_value)."." + ); + $self->logit(join("\n", @infos) . "\n", 1); + map { s/^/-- /; } @infos; + $sql_output .= "\n" . join("\n", @infos); + $sql_output .= "\n" . $fct_cost; + } + if (!$nothing) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_output); + + $self->{procedures} = (); + + my $t1 = Benchmark->new; + my $td = timediff($t1, $t0); + $self->logit("Total time to translate all functions with $num_chunk process: " . timestr($td) . "\n", 1); + + return; +} + +=head2 export_package + +Export Oracle package into PostgreSQL compatible statement. + +=cut + +sub export_package +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->{current_package} = ''; + $self->logit("Add packages definition...\n", 1); + my $nothing = 0; + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + $self->dump($sql_header); + + #--------------------------------------------------------- + # Code to use to find package parser bugs, it load a file + # containing the untouched PL/SQL code from Oracle Package + #--------------------------------------------------------- + if ($self->{input_file}) + { + $self->{plsql_pgsql} = 1; + $self->{packages} = (); + $self->logit("Reading input code from file $self->{input_file}...\n", 1); + my $content = $self->read_input_file($self->{input_file}); + my $pknm = ''; + my $before = ''; + my $old_line = ''; + my $skip_pkg_header = 0; + $self->_remove_comments(\$content); + # Normalise start of package declaration + $content =~ s/CREATE(?:\s+OR\s+REPLACE)?(?:\s+EDITIONABLE|\s+NONEDITIONABLE)?\s+PACKAGE\s+/CREATE OR REPLACE PACKAGE /igs; + # Preserve header + $content =~ s/^(.*?)(CREATE OR REPLACE PACKAGE)/$2/s; + my $start = $1 || ''; + my @pkg_content = split(/CREATE OR REPLACE PACKAGE\s+/is, $content); + for (my $i = 0; $i <= $#pkg_content; $i++) + { + # package declaration + if ($pkg_content[$i] !~ /^BODY\s+/is) + { + if ($pkg_content[$i] =~ /^([^\s]+)/is) + { + my $pname = lc($1); + $pname =~ s/"//g; + $pname =~ s/^[^\.]+\.//g; + $self->{packages}{$pname}{desc} = 'CREATE OR REPLACE PACKAGE ' . $pkg_content[$i]; + $self->{packages}{$pname}{text} = $start if ($start); + $start = ''; + } + } + else + { + if ($pkg_content[$i] =~ /^BODY\s+([^\s]+)\s+/is) + { + my $pname = lc($1); + $pname =~ s/"//g; + $pname =~ s/^[^\.]+\.//g; + $self->{packages}{$pname}{text} .= 'CREATE OR REPLACE PACKAGE ' . $pkg_content[$i]; + } + } + } + @pkg_content = (); + + foreach my $pkg (sort keys %{$self->{packages}}) + { + my $tmp_txt = ''; + if (exists $self->{packages}{$pkg}{desc}) + { + # Move comments at end of package declaration before package definition + while ($self->{packages}{$pkg}{desc} =~ s/(\%ORA2PG_COMMENT\d+\%\s*)$//) { + $self->{packages}{$pkg}{text} = $1 . $self->{packages}{$pkg}{text}; + } + } + # Get all metadata from all procedures/function when we are + # reading from a file, otherwise it has already been done + $tmp_txt = $self->{packages}{$pkg}{text}; + $tmp_txt =~ s/^.*CREATE OR REPLACE PACKAGE\s+/CREATE OR REPLACE PACKAGE /s; + my %infos = $self->_lookup_package($tmp_txt); + my $sch = 'unknown'; + my $pname = $pkg; + if ($pname =~ s/^([^\.\s]+)\.([^\s]+)$/$2/is) { + $sch = $1; + } + foreach my $f (sort keys %infos) + { + next if (!$f); + my $name = lc($f); + delete $infos{$f}{code}; + delete $infos{$f}{before}; + $pname =~ s/"//g; + $name =~ s/"//g; + %{$self->{function_metadata}{$sch}{$pname}{$name}{metadata}} = %{$infos{$f}}; + } + $self->_restore_comments(\$self->{packages}{$pkg}{text}); + } + } + + #-------------------------------------------------------- + my $default_global_vars = ''; + + my $number_fct = 0; + my $i = 1; + my $num_total_package = scalar keys %{$self->{packages}}; + foreach my $pkg (sort keys %{$self->{packages}}) + { + my $total_size = 0; + my $total_size_no_comment = 0; + my $cost_value = 0; + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i, $num_total_package, 25, '=', 'packages', "generating $pkg" ), "\r"; + } + $i++, next if (!$self->{packages}{$pkg}{text} && !$self->{packages}{$pkg}{desc}); + + # Save and cleanup previous global variables defined in other package + if (scalar keys %{$self->{global_variables}}) + { + foreach my $n (sort keys %{$self->{global_variables}}) + { + if (exists $self->{global_variables}{$n}{constant} || exists $self->{global_variables}{$n}{default}) { + $default_global_vars .= "$n = '$self->{global_variables}{$n}{default}'\n"; + } else { + $default_global_vars .= "$n = ''\n"; + } + } + } + %{$self->{global_variables}} = (); + my $pkgbody = ''; + my $fct_cost = ''; + if (!$self->{plsql_pgsql}) + { + $self->logit("Dumping package $pkg...\n", 1); + if ($self->{file_per_function}) + { + my $f = "$dirprefix\L${pkg}\E_$self->{output}"; + $f =~ s/\.(?:gz|bz2)$//i; + $pkgbody = "\\i$self->{psql_relative_path} $f\n"; + my $fhdl = $self->open_export_file("$dirprefix\L${pkg}\E_$self->{output}", 1); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->dump($sql_header . $self->{packages}{$pkg}{desc} . "\n\n" . $self->{packages}{$pkg}{text}, $fhdl); + $self->close_export_file($fhdl); + } else { + $pkgbody = $self->{packages}{$pkg}{desc} . "\n\n" . $self->{packages}{$pkg}{text}; + } + + } + else + { + $self->{current_package} = $pkg; + + # If there is a declaration only do not go further than looking at global var + if (!$self->{packages}{$pkg}{text}) + { + $self->_convert_package($pkg); + $i++; + next; + } + + if ($self->{estimate_cost}) { + $total_size += length($self->{packages}->{$pkg}{text}); + } + $self->_remove_comments(\$self->{packages}{$pkg}{text}); + + # Normalyse package creation call + $self->{packages}{$pkg}{text} =~ s/CREATE(?:\s+OR\s+REPLACE)?(?:\s+EDITIONABLE|\s+NONEDITIONABLE)?\s+PACKAGE\s+/CREATE OR REPLACE PACKAGE /is; + if ($self->{estimate_cost}) + { + my %infos = $self->_lookup_package($self->{packages}{$pkg}{text}); + foreach my $f (sort keys %infos) + { + next if (!$f); + my @cnt = $infos{$f}{code} =~ /(\%ORA2PG_COMMENT\d+\%)/i; + $total_size_no_comment += (length($infos{$f}{code}) - (17 * length(join('', @cnt)))); + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $infos{$f}{code}); + $self->logit("Function $f estimated cost: $cost\n", 1); + $cost_value += $cost; + $number_fct++; + $fct_cost .= "\t-- Function $f total estimated cost: $cost\n"; + foreach (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { + next if (!$cost_detail{$_}); + $fct_cost .= "\t\t-- $_ => $cost_detail{$_}"; + if (!$self->{is_mysql}) { + $fct_cost .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_SCORE{$_}); + } else { + $fct_cost .= " (cost: $Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_})" if ($Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE{$_}); + } + $fct_cost .= "\n"; + } + } + $cost_value += $Ora2Pg::PLSQL::OBJECT_SCORE{'PACKAGE BODY'}; + $fct_cost .= "-- Total estimated cost for package $pkg: $cost_value units, " . $self->_get_human_cost($cost_value) . "\n"; + } + $txt = $self->_convert_package($pkg); + $self->_restore_comments(\$txt) if (!$self->{file_per_function}); + $txt =~ s/(-- REVOKE ALL ON (?:FUNCTION|PROCEDURE) [^;]+ FROM PUBLIC;)/&remove_newline($1)/sge; + if (!$self->{file_per_function}) { + $self->normalize_function_call(\$txt); + } + $pkgbody .= $txt; + $pkgbody =~ s/[\r\n]*\bEND;\s*$//is; + $pkgbody =~ s/(\s*;)\s*$/$1/is; + } + if ($self->{estimate_cost}) { + $self->logit("Total size of package code: $total_size bytes.\n", 1); + $self->logit("Total size of package code without comments and header: $total_size_no_comment bytes.\n", 1); + $self->logit("Total estimated cost for package $pkg: $cost_value units, " . $self->_get_human_cost($cost_value) . ".\n", 1); + } + if ($pkgbody && ($pkgbody =~ /[a-z]/is)) { + $sql_output .= "\n\n-- Oracle package '$pkg' declaration, please edit to match PostgreSQL syntax.\n"; + $sql_output .= $pkgbody . "\n"; + $sql_output .= "-- End of Oracle package '$pkg' declaration\n\n"; + if ($self->{estimate_cost}) { + $sql_output .= "-- Total size of package code: $total_size bytes.\n"; + $sql_output .= "-- Total size of package code without comments and header: $total_size_no_comment bytes.\n"; + $sql_output .= "-- Detailed cost per function:\n" . $fct_cost; + } + $nothing++; + } + $self->{total_pkgcost} += ($number_fct*$Ora2Pg::PLSQL::OBJECT_SCORE{'FUNCTION'}); + $self->{total_pkgcost} += $Ora2Pg::PLSQL::OBJECT_SCORE{'PACKAGE BODY'}; + $i++; + } + if ($self->{estimate_cost} && $number_fct) { + $self->logit("Total number of functions found inside all packages: $number_fct.\n", 1); + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_package, 25, '=', 'packages', 'end of output.'), "\n"; + } + if (!$nothing) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_output); + + $self->{packages} = (); + $sql_output = ''; + # Create file to load custom variable initialization into postgresql.conf + if (scalar keys %{$self->{global_variables}}) { + foreach my $n (sort keys %{$self->{global_variables}}) { + if (exists $self->{global_variables}{$n}{constant} || exists $self->{global_variables}{$n}{default}) { + $default_global_vars .= "$n = '$self->{global_variables}{$n}{default}'\n"; + } else { + $default_global_vars .= "$n = ''\n"; + } + } + } + %{$self->{global_variables}} = (); + + # Save global variable that need to be initialized at startup + if ($default_global_vars) { + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + open(OUT, ">${dirprefix}global_variables.conf"); + print OUT "-- Global variables with default values used in packages.\n"; + print OUT $default_global_vars; + close(OUT); + } + + return; +} + +=head2 export_type + +Export Oracle type into PostgreSQL compatible statement. + +=cut + +sub export_type +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add custom types definition...\n", 1); + #--------------------------------------------------------- + # Code to use to find type parser issues, it load a file + # containing the untouched PL/SQL code from Oracle type + #--------------------------------------------------------- + if ($self->{input_file}) { + $self->{types} = (); + $self->logit("Reading input code from file $self->{input_file}...\n", 1); + my $content = $self->read_input_file($self->{input_file}); + $self->_remove_comments(\$content); + my $i = 0; + foreach my $l (split(/;/, $content)) { + chomp($l); + next if ($l =~ /^[\s\/]*$/s); + my $cmt = ''; + while ($l =~ s/(\%ORA2PG_COMMENT\d+\%)//s) { + $cmt .= "$1"; + } + $self->_restore_comments(\$cmt); + $l =~ s/^\s+//; + $l =~ s/^CREATE\s+(?:OR REPLACE)?\s*(?:NONEDITIONABLE|EDITIONABLE)?\s*//is; + $l .= ";\n"; + if ($l =~ /^(SUBTYPE|TYPE)\s+([^\s\(]+)/is) { + push(@{$self->{types}}, { ('name' => $2, 'code' => $l, 'comment' => $cmt, 'pos' => $i) }); + } + $i++; + } + } + #-------------------------------------------------------- + my $i = 1; + foreach my $tpe (sort {$a->{pos} <=> $b->{pos} } @{$self->{types}}) { + $self->logit("Dumping type $tpe->{name}...\n", 1); + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i, $#{$self->{types}}+1, 25, '=', 'types', "generating $tpe->{name}" ), "\r"; + } + if ($self->{plsql_pgsql}) { + $tpe->{code} = $self->_convert_type($tpe->{code}, $tpe->{owner}); + } else { + if ($tpe->{code} !~ /^SUBTYPE\s+/) { + $tpe->{code} = "CREATE$self->{create_or_replace} $tpe->{code}\n"; + } + } + $tpe->{code} =~ s/REPLACE type/REPLACE TYPE/; + $sql_output .= $tpe->{comment} . $tpe->{code} . "\n"; + $i++; + } + $self->_restore_comments(\$sql_output); + $self->{comment_values} = (); + + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $#{$self->{types}}+1, 25, '=', 'types', 'end of output.'), "\n"; + } + if (!$sql_output) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + $self->dump($sql_header . $sql_output); + + return; +} + +=head2 export_tablespace + +Export Oracle tablespace into PostgreSQL compatible statement. + +=cut + +sub export_tablespace +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + $sql_header .= "-- Oracle tablespaces export, please edit path to match your filesystem.\n"; + $sql_header .= "-- In PostgreSQl the path must be a directory and is expected to already exists\n"; + my $sql_output = ""; + + $self->logit("Add tablespaces definition...\n", 1); + + my $create_tb = ''; + my @done = (); + # Read DML from file if any + if ($self->{input_file}) { + $self->read_tablespace_from_file(); + } + my $dirprefix = ''; + foreach my $tb_type (sort keys %{$self->{tablespaces}}) { + next if ($tb_type eq 'INDEX PARTITION' || $tb_type eq 'TABLE PARTITION'); + # TYPE - TABLESPACE_NAME - FILEPATH - OBJECT_NAME + foreach my $tb_name (sort keys %{$self->{tablespaces}{$tb_type}}) { + foreach my $tb_path (sort keys %{$self->{tablespaces}{$tb_type}{$tb_name}}) { + # Replace Oracle tablespace filename + my $loc = $tb_name; + if ($tb_path =~ /^(.*[^\\\/]+)/) { + $loc = $1 . '/' . $loc; + } + if (!grep(/^$tb_name$/, @done)) { + $create_tb .= "CREATE TABLESPACE \L$tb_name\E LOCATION '$loc';\n"; + my $owner = $self->{list_tablespaces}{$tb_name}{owner} || ''; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + if ($owner) { + $create_tb .= "ALTER TABLESPACE " . $self->quote_object_name($tb_name) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + } + push(@done, $tb_name); + foreach my $obj (@{$self->{tablespaces}{$tb_type}{$tb_name}{$tb_path}}) { + next if ($self->{file_per_index} && ($tb_type eq 'INDEX')); + $sql_output .= "ALTER $tb_type " . $self->quote_object_name($obj) + . " SET TABLESPACE " . $self->quote_object_name($tb_name) . ";\n"; + } + } + } + } + + $sql_output = "$create_tb\n" . $sql_output if ($create_tb); + if (!$sql_output) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_header . $sql_output); + + if ($self->{file_per_index} && (scalar keys %{$self->{tablespaces}} > 0)) { + my $fhdl = undef; + $self->logit("Dumping tablespace alter indexes to one separate file : TBSP_INDEXES_$self->{output}\n", 1); + $fhdl = $self->open_export_file("TBSP_INDEXES_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $sql_output = ''; + foreach my $tb_type (sort keys %{$self->{tablespaces}}) { + # TYPE - TABLESPACE_NAME - FILEPATH - OBJECT_NAME + foreach my $tb_name (sort keys %{$self->{tablespaces}{$tb_type}}) { + foreach my $tb_path (sort keys %{$self->{tablespaces}{$tb_type}{$tb_name}}) { + # Replace Oracle tablespace filename + my $loc = $tb_name; + $tb_path =~ /^(.*)[^\\\/]+$/; + $loc = $1 . $loc; + foreach my $obj (@{$self->{tablespaces}{$tb_type}{$tb_name}{$tb_path}}) { + next if ($tb_type eq 'TABLE'); + $sql_output .= "ALTER $tb_type \L$obj\E SET TABLESPACE \L$tb_name\E;\n"; + } + } + } + } + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$sql_output && !$self->{no_header}); + $self->dump($sql_header . $sql_output, $fhdl); + $self->close_export_file($fhdl); + } + return; +} + +=head2 export_kettle + +Export Oracle table into Kettle script to load data into PostgreSQL. + +=cut + +sub export_kettle +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + # Remove external table from data export + if (scalar keys %{$self->{external_table}} ) { + foreach my $table (keys %{$self->{tables}}) { + if ( grep(/^$table$/i, keys %{$self->{external_table}}) ) { + delete $self->{tables}{$table}; + } + } + } + + # Ordering tables by name by default + my @ordered_tables = sort { $a cmp $b } keys %{$self->{tables}}; + if (lc($self->{data_export_order}) eq 'size') { + @ordered_tables = sort { + ($self->{tables}{$b}{table_info}{num_rows} || $self->{tables}{$a}{table_info}{num_rows}) ? + $self->{tables}{$b}{table_info}{num_rows} <=> $self->{tables}{$a}{table_info}{num_rows} : + $a cmp $b + } keys %{$self->{tables}}; + } + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + foreach my $table (@ordered_tables) { + $shell_commands .= $self->create_kettle_output($table, $dirprefix); + } + $self->dump("#!/bin/sh\n\n", $fhdl); + $self->dump("KETTLE_TEMPLATE_PATH='.'\n\n", $fhdl); + $self->dump($shell_commands, $fhdl); + + return; +} + +=head2 export_partition + +Export Oracle partition into PostgreSQL compatible statement. + +=cut + +sub export_partition +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add partitions definition...\n", 1); + + my $total_partition = 0; + foreach my $t (sort keys %{ $self->{partitions} }) { + $total_partition += scalar keys %{$self->{partitions}{$t}}; + } + foreach my $t (sort keys %{ $self->{subpartitions_list} }) + { + foreach my $p (sort keys %{ $self->{subpartitions_list}{$t} }) { + $total_partition += $self->{subpartitions_list}{$t}{$p}{count}; + } + } + + # Extract partition definition from partitioned tables + my $nparts = 1; + my $partition_indexes = ''; + foreach my $table (sort keys %{$self->{partitions}}) + { + my $function = ''; + $function = qq{ +CREATE$self->{create_or_replace} FUNCTION ${table}_insert_trigger() +RETURNS TRIGGER AS \$\$ +BEGIN +} if (!$self->{pg_supports_partition}); + + my $cond = 'IF'; + my $funct_cond = ''; + my %create_table = (); + my $idx = 0; + my $old_pos = ''; + my $old_part = ''; + my $owner = ''; + my $PGBAR_REFRESH = set_refresh_count($total_partition); + # Extract partitions in their position order + foreach my $pos (sort {$a <=> $b} keys %{$self->{partitions}{$table}}) + { + next if (!$self->{partitions}{$table}{$pos}{name}); + my $part = $self->{partitions}{$table}{$pos}{name}; + if (!$self->{quiet} && !$self->{debug} && ($nparts % $PGBAR_REFRESH) == 0) + { + print STDERR $self->progress_bar($nparts, $total_partition, 25, '=', 'partitions', "generating $table/$part" ), "\r"; + } + $nparts++; + my $create_table_tmp = ''; + my $create_table_index_tmp = ''; + my $tb_name = ''; + if ($self->{prefix_partition}) { + $tb_name = $table . "_" . $part; + } else { + if ($self->{export_schema} && !$self->{schema} && ($table =~ /^([^\.]+)\./)) { + $tb_name = $1 . '.' . $part; + } else { + $tb_name = $part; + } + } + if (!$self->{pg_supports_partition}) { + if (!exists $self->{subpartitions}{$table}{$part}) { + $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) + . " ( CHECK (\n"; + } + } else { + $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) + . " PARTITION OF \L$table\E\n"; + $create_table_tmp .= "FOR VALUES"; + } + + my @condition = (); + my @ind_col = (); + for (my $i = 0; $i <= $#{$self->{partitions}{$table}{$pos}{info}}; $i++) + { + # We received all values for partitonning on multiple column, so get the one at the right indice + my $value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$pos}{info}[$i]->{value}); + if ($#{$self->{partitions}{$table}{$pos}{info}} == 0) + { + my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$pos}{info}[$i]->{value})); + $value = $values[$i]; + } + my $old_value = ''; + if ($old_part) + { + $old_value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$old_pos}{info}[$i]->{value}); + if ($#{$self->{partitions}{$table}{$pos}{info}} == 0) + { + my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$old_pos}{info}[$i]->{value})); + $old_value = $values[$i]; + } + } + + if ($self->{partitions}{$table}{$pos}{info}[$i]->{type} eq 'LIST') + { + if (!$self->{pg_supports_partition}) { + $check_cond .= "\t$self->{partitions}{$table}{$pos}{info}[$i]->{column} IN ($value)"; + } else { + $check_cond .= " IN ($value)"; + } + } + elsif ($self->{partitions}{$table}{$pos}{info}[$i]->{type} eq 'RANGE') + { + if (!$self->{pg_supports_partition}) + { + if ($old_part eq '') { + $check_cond .= "\t$self->{partitions}{$table}{$pos}{info}[$i]->{column} < $value"; + } + else + { + $check_cond .= "\t$self->{partitions}{$table}{$pos}{info}[$i]->{column} >= $old_value" + . " AND $self->{partitions}{$table}{$pos}{info}[$i]->{column} < $value"; + } + } + else + { + if ($old_part eq '') + { + my $val = 'MINVALUE,' x ($#{$self->{partitions}{$table}{$pos}{info}}+1); + $val =~ s/,$//; + $check_cond .= " FROM ($val) TO ($value)"; + } else { + $check_cond .= " FROM ($old_value) TO ($value)"; + } + $i += $#{$self->{partitions}{$table}{$pos}{info}}; + } + } + elsif ($self->{partitions}{$table}{$pos}{info}[$i]->{type} eq 'HASH') + { + if ($self->{pg_version} < 11) + { + print STDERR "WARNING: Hash partitioning not supported, skipping partitioning of table $table\n"; + $function = ''; + $create_table_tmp = ''; + $create_table_index_tmp = ''; + next; + } + else + { + $check_cond .= " WITH (MODULUS " . (scalar keys %{$self->{partitions}{$table}}) . ", REMAINDER " . ($pos-1) . ")"; + } + } + else + { + print STDERR "WARNING: Unknown partitioning type $self->{partitions}{$table}{$pos}{info}[$i]->{type}, skipping partitioning of table $table\n"; + $create_table_tmp = ''; + $create_table_index_tmp = ''; + next; + } + if (!$self->{pg_supports_partition}) + { + $check_cond .= " AND" if ($i < $#{$self->{partitions}{$table}{$pos}{info}}); + } + my $fct = ''; + my $colname = $self->{partitions}{$table}{$pos}{info}[$i]->{column}; + if ($colname =~ s/([^\(]+)\(([^\)]+)\)/$2/) + { + $fct = $1; + } + my $cindx = $self->{partitions}{$table}{$pos}{info}[$i]->{column} || ''; + $cindx = lc($cindx) if (!$self->{preserve_case}); + $cindx = Ora2Pg::PLSQL::convert_plsql_code($self, $cindx); + my $has_hash_subpartition = 0; + if (exists $self->{subpartitions}{$table}{$part}) + { + foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part}}) + { + for (my $j = 0; $j <= $#{$self->{subpartitions}{$table}{$part}{$p}{info}}; $j++) + { + if ($self->{subpartitions}{$table}{$part}{$p}{info}[$j]->{type} eq 'HASH') + { + $has_hash_subpartition = 1; + last; + } + } + last if ($has_hash_subpartition); + } + } + + if (!exists $self->{subpartitions}{$table}{$part} || (!$self->{pg_supports_partition} && $has_hash_subpartition)) + { + # Reproduce indexes definition from the main table before PG 11 + # after they are automatically created on partition tables + if ($self->{pg_version} < 11) + { + my ($idx, $fts_idx) = $self->_create_indexes($table, 0, %{$self->{tables}{$table}{indexes}}); + my $tb_name2 = $self->quote_object_name($tb_name); + $create_table_index_tmp .= "CREATE INDEX " + . $self->quote_object_name("${tb_name}_$colname$pos") + . " ON " . $self->quote_object_name($tb_name) . " ($cindx);\n"; + if ($idx || $fts_idx) + { + $idx =~ s/ $table/ $tb_name2/igs; + $fts_idx =~ s/ $table/ $tb_name2/igs; + # remove indexes already created + $idx =~ s/CREATE [^;]+ \($cindx\);//; + $fts_idx =~ s/CREATE [^;]+ \($cindx\);//; + if ($idx || $fts_idx) + { + # fix index name to avoid duplicate index name + $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1$pos /gs; + $fts_idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1$pos /gs; + $create_table_index_tmp .= "-- Reproduce partition indexes that was defined on the parent table\n"; + } + $create_table_index_tmp .= "$idx\n" if ($idx); + $create_table_index_tmp .= "$fts_idx\n" if ($fts_idx); + } + + # Set the unique (and primary) key definition + $idx = $self->_create_unique_keys($table, $self->{tables}{$table}{unique_key}); + if ($idx) + { + $idx =~ s/ $table/ $tb_name2/igs; + # remove indexes already created + $idx =~ s/CREATE [^;]+ \($cindx\);//; + if ($idx) + { + # fix index name to avoid duplicate index name + $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1$pos /gs; + $create_table_index_tmp .= "-- Reproduce partition unique indexes / pk that was defined on the parent table\n"; + $create_table_index_tmp .= "$idx\n"; + # Remove duplicate index with this one + if ($idx =~ /ALTER TABLE $tb_name2 ADD PRIMARY KEY (.*);/s) + { + my $collist = quotemeta($1); + $create_table_index_tmp =~ s/CREATE INDEX [^;]+ ON $tb_name2 $collist;//s; + } + } + } + } + } + my $deftb = ''; + $deftb = "${table}_" if ($self->{prefix_partition}); + if ($self->{partitions_default}{$table} && ($create_table{$table}{index} !~ /ON $deftb$self->{partitions_default}{$table} /)) + { + $cindx = $self->{partitions}{$table}{$pos}{info}[$i]->{column} || ''; + $cindx = lc($cindx) if (!$self->{preserve_case}); + $cindx = Ora2Pg::PLSQL::convert_plsql_code($self, $cindx); + $create_table_index_tmp .= "CREATE INDEX " . $self->quote_object_name("$deftb$self->{partitions_default}{$table}_$colname") . " ON " . $self->quote_object_name("$deftb$self->{partitions_default}{$table}") . " ($cindx);\n"; + } + push(@ind_col, $self->{partitions}{$table}{$pos}{info}[$i]->{column}) if (!grep(/^$self->{partitions}{$table}{$pos}{info}[$i]->{column}$/, @ind_col)); + if ($self->{partitions}{$table}{$pos}{info}[$i]->{type} eq 'LIST') + { + if (!$fct) { + push(@condition, "NEW.$self->{partitions}{$table}{$pos}{info}[$i]->{column} IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$pos}{info}[$i]->{value}) . ")"); + } else { + push(@condition, "$fct(NEW.$colname) IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$pos}{info}[$i]->{value}) . ")"); + } + } + elsif ($self->{partitions}{$table}{$pos}{info}[$i]->{type} eq 'RANGE') + { + if (!$fct) { + push(@condition, "NEW.$self->{partitions}{$table}{$pos}{info}[$i]->{column} < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$pos}{info}[$i]->{value})); + } else { + push(@condition, "$fct(NEW.$colname) < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{partitions}{$table}{$pos}{info}[$i]->{value})); + } + } + $owner = $self->{partitions}{$table}{$pos}{info}[$i]->{owner} || ''; + } + + if (!$self->{pg_supports_partition}) + { + if ($self->{partitions}{$table}{$pos}{info}[$i]->{type} ne 'HASH') + { + if (!exists $self->{subpartitions}{$table}{$part}) + { + $create_table_tmp .= $check_cond . "\n"; + $create_table_tmp .= ") ) INHERITS ($table);\n"; + } + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + if ($owner) { + $create_table_tmp .= "ALTER TABLE " . $self->quote_object_name($tb_name) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + } + } + else + { + $create_table_tmp .= $check_cond; + if (exists $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{type}) + { + $create_table_tmp .= "\nPARTITION BY " . $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{type} . " ("; + for (my $j = 0; $j <= $#{$self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{columns}}; $j++) + { + $create_table_tmp .= ', ' if ($j > 0); + $create_table_tmp .= $self->quote_object_name($self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{columns}[$j]); + } + $create_table_tmp .= ")"; + } + $create_table_tmp .= ";\n"; + } + # Add subpartition if any defined on Oracle + my $sub_funct_cond = ''; + my $sub_old_part = ''; + if (exists $self->{subpartitions}{$table}{$part}) + { + my $sub_cond = 'IF'; + my $sub_funct_cond_tmp = ''; + my $create_subtable_tmp = ''; + my $total_subpartition = scalar %{$self->{subpartitions}{$table}{$part}} || 0; + foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part}}) + { + my $subpart = $self->{subpartitions}{$table}{$part}{$p}{name}; + my $sub_tb_name = $subpart; + $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any + if ($self->{prefix_partition}) + { + if ($self->{prefix_sub_partition}) { + $sub_tb_name = "${tb_name}_$sub_tb_name"; + } else { + $sub_tb_name = "${table}_$sub_tb_name"; + } + } + if (!$self->{quiet} && !$self->{debug} && ($nparts % $PGBAR_REFRESH) == 0) + { + print STDERR $self->progress_bar($nparts, $total_partition, 25, '=', 'partitions', "generating $table/$part/$subpart" ), "\r"; + } + $nparts++; + $create_subtable_tmp .= "CREATE TABLE " . $self->quote_object_name($sub_tb_name); + if (!$self->{pg_supports_partition}) { + $create_subtable_tmp .= " ( CHECK (\n"; + } + else + { + $create_subtable_tmp .= " PARTITION OF " . $self->quote_object_name($tb_name) . "\n"; + $create_subtable_tmp .= "FOR VALUES"; + } + my $sub_check_cond_tmp = ''; + my @subcondition = (); + for (my $i = 0; $i <= $#{$self->{subpartitions}{$table}{$part}{$p}{info}}; $i++) + { + # We received all values for partitonning on multiple column, so get the one at the right indice + my $value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value}); + if ($#{$self->{subpartitions}{$table}{$part}{$p}{info}} == 0) + { + my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value})); + $value = $values[$i]; + } + my $old_value = ''; + if ($sub_old_part) + { + $old_value = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[$i]->{value}); + if ($#{$self->{subpartitions}{$table}{$part}{$p}{info}} == 0) + { + my @values = split(/,\s/, Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[$i]->{value})); + $old_value = $values[$i]; + } + } + + if ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'LIST') + { + if (!$self->{pg_supports_partition}) { + $sub_check_cond_tmp .= "$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} IN ($value)"; + } else { + $sub_check_cond_tmp .= " IN ($value)"; + } + } + elsif ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'RANGE') + { + if (!$self->{pg_supports_partition}) + { + if ($old_part eq '') { + $sub_check_cond_tmp .= "\t$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} < $value"; + } + else + { + $sub_check_cond_tmp .= "\t$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} >= $old_value" + . " AND $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} < $value"; + } + } + else + { + if ($old_part eq '') + { + my $val = 'MINVALUE,' x ($#{$self->{subpartitions}{$table}{$part}{$p}{info}}+1); + $val =~ s/,$//; + $sub_check_cond_tmp .= " FROM ($val) TO ($value)"; + } else { + $sub_check_cond_tmp .= " FROM ($old_value) TO ($value)"; + } + $i += $#{$self->{subpartitions}{$table}{$part}{$p}{info}}; + } + } + elsif ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'HASH') + { + if ($self->{pg_version} < 11) + { + print STDERR "WARNING: Hash partitioning not supported, skipping subpartitioning of table $table\n"; + $create_subtable_tmp = ''; + $sub_funct_cond_tmp = ''; + next; + } else { + $sub_check_cond_tmp .= " WITH (MODULUS " . $self->{subpartitions_list}{"\L$table\E"}{"\L$part\E"}{count} . ", REMAINDER " . ($p-1) . ")"; + } + } + else + { + print STDERR "WARNING: Unknown partitioning type $self->{partitions}{$table}{$pos}{info}[$i]->{type}, skipping partitioning of table $table\n"; + $create_subtable_tmp = ''; + $sub_funct_cond_tmp = ''; + next; + } + if (!$self->{pg_supports_partition}) { + $sub_check_cond_tmp .= " AND " if ($i < $#{$self->{subpartitions}{$table}{$part}{$p}{info}}); + } + # Reproduce indexes definition from the main table before PG 11 + # after they are automatically created on partition tables + if ($self->{pg_version} < 11) + { + push(@ind_col, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column}) if (!grep(/^$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column}$/, @ind_col)); + my $fct = ''; + my $colname = $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column}; + if ($colname =~ s/([^\(]+)\(([^\)]+)\)/$2/) { + $fct = $1; + } + $cindx = join(',', @ind_col); + $cindx = lc($cindx) if (!$self->{preserve_case}); + $cindx = Ora2Pg::PLSQL::convert_plsql_code($self, $cindx); + $create_table_index_tmp .= "CREATE INDEX " . $self->quote_object_name("${sub_tb_name}_$colname$p") + . " ON " . $self->quote_object_name("$sub_tb_name") . " ($cindx);\n"; + my $tb_name2 = $self->quote_object_name("$sub_tb_name"); + # Reproduce indexes definition from the main table + my ($idx, $fts_idx) = $self->_create_indexes($table, 0, %{$self->{tables}{$table}{indexes}}); + if ($idx || $fts_idx) { + $idx =~ s/ $table/ $tb_name2/igs; + $fts_idx =~ s/ $table/ $tb_name2/igs; + # remove indexes already created + $idx =~ s/CREATE [^;]+ \($cindx\);//; + $fts_idx =~ s/CREATE [^;]+ \($cindx\);//; + if ($idx || $fts_idx) { + # fix index name to avoid duplicate index name + $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1${pos}_$p /gs; + $fts_idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1${pos}_$p /gs; + $create_table_index_tmp .= "-- Reproduce subpartition indexes that was defined on the parent table\n"; + } + $create_table_index_tmp .= "$idx\n" if ($idx); + $create_table_index_tmp .= "$fts_idx\n" if ($fts_idx); + } + + # Set the unique (and primary) key definition + $idx = $self->_create_unique_keys($table, $self->{tables}{$table}{unique_key}); + if ($idx) { + $create_table_index_tmp .= "-- Reproduce subpartition unique indexes / pk that was defined on the parent table\n"; + $idx =~ s/ $table/ $tb_name2/igs; + # remove indexes already created + $idx =~ s/CREATE [^;]+ \($cindx\);//; + if ($idx) { + # fix index name to avoid duplicate index name + $idx =~ s/(CREATE(?:.*?)INDEX ([^\s]+)) /$1${pos}_$p /gs; + $create_table_index_tmp .= "$idx\n"; + # Remove duplicate index with this one + if ($idx =~ /ALTER TABLE $tb_name2 ADD PRIMARY KEY (.*);/s) { + my $collist = quotemeta($1); + $create_table_index_tmp =~ s/CREATE INDEX [^;]+ ON $tb_name2 $collist;//s; + } + } + } + } + if ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'LIST') { + if (!$fct) { + push(@subcondition, "NEW.$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value}) . ")"); + } else { + push(@subcondition, "$fct(NEW.$colname) IN (" . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value}) . ")"); + } + } elsif ($self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{type} eq 'RANGE') { + if (!$fct) { + push(@subcondition, "NEW.$self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{column} < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value})); + } else { + push(@subcondition, "$fct(NEW.$colname) < " . Ora2Pg::PLSQL::convert_plsql_code($self, $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{value})); + } + } + $owner = $self->{subpartitions}{$table}{$part}{$p}{info}[$i]->{owner} || ''; + } + if ($self->{pg_supports_partition}) { + $sub_check_cond_tmp .= ';'; + $create_subtable_tmp .= "$sub_check_cond_tmp\n"; + } else { + $create_subtable_tmp .= $check_cond; + $create_subtable_tmp .= " AND $sub_check_cond_tmp" if ($sub_check_cond_tmp); + $create_subtable_tmp .= "\n) ) INHERITS ($table);\n"; + } + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + if ($owner) { + $create_subtable_tmp .= "ALTER TABLE " . $self->quote_object_name("$sub_tb_name") + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + if ($#subcondition >= 0) { + $sub_funct_cond_tmp .= "\t\t$sub_cond ( " . join(' AND ', @subcondition) . " ) THEN INSERT INTO " + . $self->quote_object_name("$sub_tb_name") . " VALUES (NEW.*);\n"; + $sub_cond = 'ELSIF'; + } + $sub_old_part = $part; + $sub_old_pos = $p; + } + if ($create_subtable_tmp) { + $create_table_tmp .= $create_subtable_tmp; + $sub_funct_cond = $sub_funct_cond_tmp; + } + } + $check_cond = ''; + + if ($#condition >= 0) + { + if (!$sub_funct_cond) { + $funct_cond .= "\t$cond ( " . join(' AND ', @condition) . " ) THEN INSERT INTO " . $self->quote_object_name($tb_name) . " VALUES (NEW.*);\n"; + } + else + { + my $sub_old_pos = 0; + if (!$self->{pg_supports_partition}) + { + $sub_funct_cond = Ora2Pg::PLSQL::convert_plsql_code($self, $sub_funct_cond); + $funct_cond .= "\t$cond ( " . join(' AND ', @condition) . " ) THEN \n"; + $funct_cond .= $sub_funct_cond; + if (exists $self->{subpartitions_default}{$table}{$part}) + { + my $deftb = ''; + $deftb = "${table}_" if ($self->{prefix_partition}); + $funct_cond .= "\t\tELSE INSERT INTO " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}") + . " VALUES (NEW.*);\n\t\tEND IF;\n"; + $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}") + . " () INHERITS ($table);\n"; + $create_table_index_tmp .= "CREATE INDEX " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}_$pos") + . " ON " . $self->quote_object_name("$deftb$self->{subpartitions_default}{$table}{$part}") . " ($cindx);\n"; + } + else + { + $funct_cond .= qq{ ELSE + -- Raise an exception + RAISE EXCEPTION 'Value out of range in subpartition. Fix the ${table}_insert_trigger() function!'; + }; + $funct_cond .= "\t\tEND IF;\n"; + } + + # With default partition just add default and continue + } + elsif (exists $self->{subpartitions_default}{$table}{$part}) + { + my $tb_name = $self->{subpartitions_default}{$table}{$part}; + if ($self->{prefix_partition}) { + $tb_name = $table . "_" . $self->{subpartitions_default}{$table}{$part}; + } elsif ($self->{export_schema} && !$self->{schema} && ($table =~ /^([^\.]+)\./)) { + $tb_name = $1 . '.' . $self->{subpartitions_default}{$table}{$part}; + } + if ($self->{pg_version} >= 11) { + $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) + . " PARTITION OF \L$table\E DEFAULT;\n"; + } elsif ($self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[$i]->{type} eq 'RANGE') { + $create_table_tmp .= "CREATE TABLE " . $self->quote_object_name($tb_name) + . " PARTITION OF \L$table\E FOR VALUES FROM ($self->{subpartitions}{$table}{$part}{$sub_old_pos}{info}[-1]->{value}) TO (MAX_VALUE);\n"; + } + } + } + $cond = 'ELSIF'; + } + $old_part = $part; + $old_pos = $pos; + $create_table{$table}{table} .= $create_table_tmp; + $create_table{$table}{index} .= $create_table_index_tmp; + } + + if (exists $create_table{$table}) + { + if (!$self->{pg_supports_partition}) + { + if ($self->{partitions_default}{$table}) + { + my $deftb = ''; + $deftb = "${table}_" if ($self->{prefix_partition}); + my $pname = $self->quote_object_name("$deftb$self->{partitions_default}{$table}"); + $function .= $funct_cond . qq{ ELSE + INSERT INTO $pname VALUES (NEW.*); +}; + } + elsif ($function) + { + $function .= $funct_cond . qq{ ELSE + -- Raise an exception + RAISE EXCEPTION 'Value out of range in partition. Fix the ${table}_insert_trigger() function!'; +}; + } + $function .= qq{ +END IF; +RETURN NULL; +END; +\$\$ +LANGUAGE plpgsql; +} if ($function); + $function = Ora2Pg::PLSQL::convert_plsql_code($self, $function); + } + else + { + # With default partition just add default and continue + if (exists $self->{partitions_default}{$table}) + { + my $tb_name = ''; + if ($self->{prefix_partition}) { + $tb_name = $table . "_" . $self->{partitions_default}{$table}; + } + else + { + if ($self->{export_schema} && !$self->{schema} && ($table =~ /^([^\.]+)\./)) { + $tb_name = $1 . '.' . $self->{partitions_default}{$table}; + } else { + $tb_name = $self->{partitions_default}{$table}; + } + } + if ($self->{pg_version} >= 11) { + $create_table{$table}{table} .= "CREATE TABLE " . $self->quote_object_name($tb_name) + . " PARTITION OF \L$table\E DEFAULT;\n"; + } else { + $create_table{$table}{table} .= "CREATE TABLE " . $self->quote_object_name($tb_name) + . " PARTITION OF \L$table\E FOR VALUES FROM ($self->{partitions}{$table}{$old_pos}{info}[-1]->{value}) TO (MAX_VALUE);\n"; + } + } + } + } + + if (exists $create_table{$table}) + { + $partition_indexes .= qq{ +-- Create indexes on each partition of table $table +$create_table{$table}{'index'} + +} if ($create_table{$table}{'index'}); + $sql_output .= qq{ +$create_table{$table}{table} +}; + my $tb = $self->quote_object_name($table); + my $trg = $self->quote_object_name("${table}_insert_trigger"); + my $defname = $self->{partitions_default}{$table}; + $defname = $table . '_' . $defname if ($self->{prefix_partition}); + $defname = $self->quote_object_name($defname); + if (!$self->{pg_supports_partition} && $function) + { + $sql_output .= qq{ +-- Create default table, where datas are inserted if no condition match +CREATE TABLE $defname () INHERITS ($tb); +} if ($self->{partitions_default}{$table}); + $sql_output .= qq{ + +$function + +CREATE TRIGGER ${table}_trigger_insert +BEFORE INSERT ON $table +FOR EACH ROW EXECUTE PROCEDURE $trg(); + +------------------------------------------------------------------------------- +}; + + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + if ($owner) + { + $sql_output .= "ALTER TABLE " . $self->quote_object_name($self->{partitions_default}{$table}) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n" + if ($self->{partitions_default}{$table}); + $sql_output .= "ALTER FUNCTION " . $self->quote_object_name("${table}_insert_trigger") + . "() OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + } + } + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($nparts - 1, $total_partition, 25, '=', 'partitions', 'end of output.'), "\n"; + } + if (!$sql_output) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + $self->dump($sql_header . $sql_output); + + if ($partition_indexes) + { + my $fhdl = undef; + $self->logit("Dumping partition indexes to file : PARTITION_INDEXES_$self->{output}\n", 1); + $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; + $sql_header .= "-- Copyright 2000-2020 Gilles DAROLD. All rights reserved.\n"; + $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; + $sql_header = '' if ($self->{no_header}); + $fhdl = $self->open_export_file("PARTITION_INDEXES_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->dump($sql_header . $partition_indexes, $fhdl); + $self->close_export_file($fhdl); + } + + return; +} + +=head2 export_synonym + +Export Oracle synonym into PostgreSQL compatible statement. + +=cut + +sub export_synonym +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Add synonyms definition...\n", 1); + # Read DML from file if any + if ($self->{input_file}) { + $self->read_synonym_from_file(); + } + my $i = 1; + my $num_total_synonym = scalar keys %{$self->{synonyms}}; + my $count_syn = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_synonym); + foreach my $syn (sort { $a cmp $b } keys %{$self->{synonyms}}) + { + if (!$self->{quiet} && !$self->{debug} && ($count_syn % $PGBAR_REFRESH) == 0) { + print STDERR $self->progress_bar($i, $num_total_synonym, 25, '=', 'synonyms', "generating $syn" ), "\r"; + } + $count_syn++; + if ($self->{synonyms}{$syn}{dblink}) { + $sql_output .= "-- You need to create foreign table $self->{synonyms}{$syn}{table_owner}.$self->{synonyms}{$syn}{table_name} using foreign server: $self->{synonyms}{$syn}{dblink} (see DBLINK and FDW export type)\n"; + } + $sql_output .= "CREATE VIEW " . $self->quote_object_name("$self->{synonyms}{$syn}{owner}.$syn") + . " AS SELECT * FROM " . $self->quote_object_name("$self->{synonyms}{$syn}{table_owner}.$self->{synonyms}{$syn}{table_name}") . ";\n"; + my $owner = $self->{synonyms}{$syn}{table_owner}; + $owner = $self->{force_owner} if ($self->{force_owner} && ($self->{force_owner} ne "1")); + $sql_output .= "ALTER VIEW " . $self->quote_object_name("$self->{synonyms}{$syn}{owner}.$syn") + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + $sql_output .= "GRANT ALL ON " . $self->quote_object_name("$self->{synonyms}{$syn}{owner}.$syn") + . " TO " . $self->quote_object_name($self->{synonyms}{$syn}{owner}) . ";\n\n"; + $i++; + } + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($i - 1, $num_total_synonym, 25, '=', 'synonyms', 'end of output.'), "\n"; + } + if (!$sql_output) { + $sql_output = "-- Nothing found of type $self->{type}\n" if (!$self->{no_header}); + } + + $self->dump($sql_header . $sql_output); + + return; +} + +=head2 export_table + +Export Oracle table into PostgreSQL compatible statement. + +=cut + +sub export_table +{ + my $self = shift; + + my $sql_header = $self->_set_file_header(); + my $sql_output = ""; + + $self->logit("Exporting tables...\n", 1); + + if ($self->{export_schema} && ($self->{schema} || $self->{pg_schema})) + { + if ($self->{create_schema}) { + if ($self->{pg_schema} && $self->{pg_schema} =~ /,/) { + $self->logit("FATAL: with export type TABLE you can not set multiple schema to PG_SCHEMA when EXPORT_SCHEMA is enabled.\n", 0, 1); + } + $sql_output .= "CREATE SCHEMA IF NOT EXISTS " . $self->quote_object_name($self->{pg_schema} || $self->{schema}) . ";\n"; + } + my $owner = ''; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $owner ||= $self->{schema}; + if ($owner && $self->{create_schema}) { + $sql_output .= "ALTER SCHEMA " . $self->quote_object_name($self->{pg_schema} || $self->{schema}) . " OWNER TO \L$owner\E;\n"; + } + $sql_output .= "\n"; + } + elsif ($self->{export_schema}) + { + if ($self->{create_schema}) { + my $current_schema = ''; + foreach my $table (sort keys %{$self->{tables}}) { + if ($table =~ /^([^\.]+)\..*/) { + if ($1 ne $current_schema) { + $current_schema = $1; + $sql_output .= "CREATE SCHEMA IF NOT EXISTS " . $self->quote_object_name($1) . ";\n"; + } + } + } + } + } + $sql_output .= $self->set_search_path(); + + # Read DML from file if any + if ($self->{input_file}) { + $self->read_schema_from_file(); + } + + my $constraints = ''; + if ($self->{file_per_constraint}) { + $constraints .= $self->set_search_path(); + } + my $indices = ''; + my $fts_indices = ''; + + # Find first the total number of tables + my $num_total_table = scalar keys %{$self->{tables}}; + + # Hash that will contains virtual column information to build triggers + my %virtual_trigger_info = (); + + # Stores DDL to restart autoincrement sequences + my $sequence_output = ''; + + # Dump all table/index/constraints SQL definitions + my $ib = 1; + my $count_table = 0; + my $PGBAR_REFRESH = set_refresh_count($num_total_table); + foreach my $table (sort { + if (exists $self->{tables}{$a}{internal_id}) { + $self->{tables}{$a}{internal_id} <=> $self->{tables}{$b}{internal_id}; + } else { + $a cmp $b; + } + } keys %{$self->{tables}}) + { + # Foreign table can not be temporary + next if ($self->{type} eq 'FDW' and $self->{tables}{$table}{table_info}{type} =~/ TEMPORARY/); + + $self->logit("Dumping table $table...\n", 1); + if (!$self->{quiet} && !$self->{debug} && ($count_table % $PGBAR_REFRESH) == 0) { + print STDERR $self->progress_bar($ib, $num_total_table, 25, '=', 'tables', "exporting $table" ), "\r"; + } + $count_table++; + + # Create FDW server if required + if ($self->{external_to_fdw}) { + if ( grep(/^$table$/i, keys %{$self->{external_table}}) ) { + $sql_header .= "CREATE EXTENSION IF NOT EXISTS file_fdw;\n\n" if ($sql_header !~ /CREATE EXTENSION .* file_fdw;/is); + $sql_header .= "CREATE SERVER \L$self->{external_table}{$table}{directory}\E FOREIGN DATA WRAPPER file_fdw;\n\n" if ($sql_header !~ /CREATE SERVER $self->{external_table}{$table}{directory} FOREIGN DATA WRAPPER file_fdw;/is); + } + } + + my $tbname = $self->get_replaced_tbname($table); + my $foreign = ''; + if ( ($self->{type} eq 'FDW') || ($self->{external_to_fdw} && (grep(/^$table$/i, keys %{$self->{external_table}}) || $self->{tables}{$table}{table_info}{connection})) ) { + $foreign = ' FOREIGN'; + } + my $obj_type = $self->{tables}{$table}{table_info}{type} || 'TABLE'; + if ( ($obj_type eq 'TABLE') && $self->{tables}{$table}{table_info}{nologging} && !$self->{disable_unlogged} ) { + $obj_type = 'UNLOGGED ' . $obj_type; + } + if (exists $self->{tables}{$table}{table_as}) { + if ($self->{plsql_pgsql}) { + $self->{tables}{$table}{table_as} = Ora2Pg::PLSQL::convert_plsql_code($self, $self->{tables}{$table}{table_as}); + } + my $withoid = _make_WITH($self->{with_oid}, $self->{tables}{$tbname}{table_info}); + $sql_output .= "\nCREATE $obj_type $tbname $withoid AS $self->{tables}{$table}{table_as};\n"; + next; + } + if (exists $self->{tables}{$table}{truncate_table}) { + $sql_output .= "\nTRUNCATE TABLE $tbname;\n"; + } + my $serial_sequence = ''; + my $enum_str = ''; + my @skip_column_check = (); + if (exists $self->{tables}{$table}{column_info}) { + my $schem = ''; + $sql_output .= "\nCREATE$foreign $obj_type $tbname (\n"; + + # Extract column information following the Oracle position order + foreach my $k (sort { + if (!$self->{reordering_columns}) { + $self->{tables}{$table}{column_info}{$a}[11] <=> $self->{tables}{$table}{column_info}{$b}[11]; + } else { + my $tmpa = $self->{tables}{$table}{column_info}{$a}; + $tmpa->[2] =~ s/\D//g; + my $typa = $self->_sql_type($tmpa->[1], $tmpa->[2], $tmpa->[5], $tmpa->[6], $tmpa->[4]); + $typa =~ s/\(.*//; + my $tmpb = $self->{tables}{$table}{column_info}{$b}; + $tmpb->[2] =~ s/\D//g; + my $typb = $self->_sql_type($tmpb->[1], $tmpb->[2], $tmpb->[5], $tmpb->[6], $tmpb->[4]); + $typb =~ s/\(.*//; + if($TYPALIGN{$typa} != $TYPALIGN{$typb}){ + # sort by field size asc + $TYPALIGN{$typa} <=> $TYPALIGN{$typb}; + }else{ + # if same size sort by original position + $self->{tables}{$table}{column_info}{$a}[11] <=> $self->{tables}{$table}{column_info}{$b}[11]; + } + } + } keys %{$self->{tables}{$table}{column_info}}) { + + # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE + my $f = $self->{tables}{$table}{column_info}{$k}; + $f->[2] =~ s/\D//g; + my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4]); + $type = "$f->[1], $f->[2]" if (!$type); + # Change column names + my $fname = $f->[0]; + if (exists $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"} && $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"}) { + $self->logit("\tReplacing column \L$f->[0]\E as " . $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"} . "...\n", 1); + $fname = $self->{replaced_cols}{"\L$table\E"}{"\L$fname\E"}; + } + + # Check if we need auto increment + if ($f->[12] eq 'auto_increment' || $f->[12] eq '1') { + if ($type !~ s/bigint/bigserial/) { + if ($type !~ s/smallint/smallserial/) { + $type =~ s/integer/serial/; + } + } + if ($type =~ /serial/) { + my $seqname = lc($tbname) . '_' . lc($fname) . '_seq'; + if ($self->{preserve_case}) { + $seqname = $tbname . '_' . $fname . '_seq'; + } + my $tobequoted = 0; + if ($seqname =~ s/"//g) { + $tobequoted = 1; + } + + if (length($seqname) > 63) { + if (length($tbname) > 29) { + $seqname = substr(lc($tbname), 0, 29); + } else { + $seqname = lc($tbname); + } + if (length($fname) > 29) { + $seqname .= '_' . substr(lc($fname), 0, 29); + } else { + $seqname .= '_' . lc($fname); + } + $seqname .= '_seq'; + } + if ($tobequoted) { + $seqname = '"' . $seqname . '"'; + } + $serial_sequence .= "ALTER SEQUENCE $seqname RESTART WITH $self->{tables}{$table}{table_info}{auto_increment};\n" if (exists $self->{tables}{$table}{table_info}{auto_increment}); + } + } + + # Check if this column should be replaced by a boolean following table/column name + if ($f->[1] =~ /ENUM/i) { + $f->[1] =~ s/^ENUM\(//i; + $f->[1] =~ s/\)$//; + my $keyname = $tbname . '_' . $fname . '_chk'; + $keyname =~ s/(.*)"(_${fname}_chk)/$1$2"/; # used when preserve_case is enable + $enum_str .= "ALTER TABLE $tbname ADD CONSTRAINT $keyname CHECK ($fname IN ($f->[1]));\n"; + $type = 'varchar'; + } + my $typlen = $f->[5]; + $typlen ||= $f->[2]; + if (grep(/^$f->[0]$/i, @{$self->{'replace_as_boolean'}{uc($table)}})) { + $type = 'boolean'; + push(@skip_column_check, $fname); + # Check if this column should be replaced by a boolean following type/precision + } elsif (exists $self->{'replace_as_boolean'}{uc($f->[1])} && ($self->{'replace_as_boolean'}{uc($f->[1])}[0] == $typlen)) { + $type = 'boolean'; + push(@skip_column_check, $fname); + } + if ($f->[1] =~ /SDO_GEOMETRY/) { + # 12:SRID,13:SDO_DIM,14:SDO_GTYPE + # Set the dimension, array is (srid, dims, gtype) + my $suffix = ''; + if ($f->[13] == 3) { + $suffix = 'Z'; + } elsif ($f->[13] == 4) { + $suffix = 'ZM'; + } + my $gtypes = ''; + if (!$f->[14] || ($f->[14] =~ /,/) ) { + $gtypes = $ORA2PG_SDO_GTYPE{0}; + } else { + $gtypes = $f->[14]; + } + $type = "geometry($gtypes$suffix"; + if ($f->[12]) { + $type .= ",$f->[12]"; + } + $type .= ")"; + } + $type = $self->{'modify_type'}{"\L$table\E"}{"\L$f->[0]\E"} if (exists $self->{'modify_type'}{"\L$table\E"}{"\L$f->[0]\E"}); + $fname = $self->quote_object_name($fname); + $sql_output .= "\t$fname $type"; + if ($foreign && $self->is_primary_key_column($table, $f->[0])) { + $sql_output .= " OPTIONS (key 'true')"; + } + if (!$f->[3] || ($f->[3] =~ /^N/)) { + # smallserial, serial and bigserial use a NOT NULL sequence as default value, + # so we don't need to add it here + if ($type !~ /serial/) { + push(@{$self->{tables}{$table}{check_constraint}{notnull}}, $f->[0]); + $sql_output .= " NOT NULL"; + } + } + + # Autoincremented columns + if (!$self->{schema} && $self->{export_schema}) { + $f->[8] = "$f->[9].$f->[8]"; + } + if (exists $self->{identity_info}{$f->[8]}{$f->[0]} and $self->{type} ne 'FDW') + { + $sql_output =~ s/ NOT NULL\s*$//s; # IDENTITY or serial column are NOT NULL by default + if ($self->{pg_supports_identity}) + { + $sql_output =~ s/ [^\s]+$/ bigint/; # Force bigint + $sql_output .= " GENERATED $self->{identity_info}{$f->[8]}{$f->[0]}{generation} AS IDENTITY"; + $sql_output .= " (" . $self->{identity_info}{$f->[8]}{$f->[0]}{options} . ')' if (exists $self->{identity_info}{$f->[8]}{$f->[0]}{options} && $self->{identity_info}{$f->[8]}{$f->[0]}{options} ne ''); + } + else + { + $sql_output =~ s/bigint\s*$/bigserial/s; + $sql_output =~ s/smallint\s*$/smallserial/s; + $sql_output =~ s/(integer|int)\s*$/serial/s; + } + $sql_output .= ",\n"; + $sequence_output .= "SELECT ora2pg_upd_autoincrement_seq('$f->[8]','$f->[0]');\n"; + next; + } + + # Default value + if ($f->[4] ne "" && uc($f->[4]) ne 'NULL') + { + $f->[4] =~ s/^\s+//; + $f->[4] =~ s/\s+$//; + $f->[4] =~ s/"//gs; + if ($self->{plsql_pgsql}) { + $f->[4] = Ora2Pg::PLSQL::convert_plsql_code($self, $f->[4]); + } + # Check if this is a virtual column before proceeding to default value export + if ($self->{tables}{$table}{column_info}{$k}[10] eq 'YES') { + $virtual_trigger_info{$table}{$k} = $f->[4]; + $virtual_trigger_info{$table}{$k} =~ s/"//gs; + foreach my $c (keys %{$self->{tables}{$table}{column_info}}) { + $virtual_trigger_info{$table}{$k} =~ s/\b$c\b/NEW.$c/gs; + } + + } else { + + if (($f->[4] ne '') && ($self->{type} ne 'FDW')) { + if ($type eq 'boolean') { + my $found = 0; + foreach my $k (sort {$b cmp $a} %{ $self->{ora_boolean_values} }) { + if ($f->[4] =~ /\b$k\b/i) { + $sql_output .= " DEFAULT '" . $self->{ora_boolean_values}{$k} . "'"; + $found = 1; + last; + } + } + $sql_output .= " DEFAULT " . $f->[4] if (!$found); + } else { + if (($f->[4] !~ /^'/) && ($f->[4] =~ /[^\d\.]/)) { + if ($type =~ /CHAR|TEXT|ENUM/i) { + $f->[4] = "'$f->[4]'" if ($f->[4] !~ /[']/ && $f->[4] !~ /\(.*\)/); + } elsif ($type =~ /DATE|TIME/i) { + if ($f->[4] =~ /0000-00-00/) { + if ($self->{replace_zero_date}) { + $f->[4] = $self->{replace_zero_date}; + } else { + $f->[4] =~ s/^0000-00-00/1970-01-01/; + } + } + if ($f->[4] =~ /^\d+/) { + $f->[4] = "'$f->[4]'"; + } elsif ($f->[4] =~ /^[\-]*INFINITY$/) { + $f->[4] = "'$f->[4]'::$type"; + } elsif ($f->[4] =~ /AT TIME ZONE/i) { + $f->[4] = "($f->[4])"; + } + } + } + else + { + my @c = $f->[4] =~ /\./g; + if ($#c >= 1) + { + if ($type =~ /CHAR|TEXT|ENUM/i) { + $f->[4] = "'$f->[4]'" if ($f->[4] !~ /[']/ && $f->[4] !~ /\(.*\)/); + } elsif ($type =~ /DATE|TIME/i) { + if ($f->[4] =~ /0000-00-00/) { + if ($self->{replace_zero_date}) { + $f->[4] = $self->{replace_zero_date}; + } else { + $f->[4] =~ s/^0000-00-00/1970-01-01/; + } + } + if ($f->[4] =~ /^\d+/) { + $f->[4] = "'$f->[4]'"; + } elsif ($f->[4] =~ /^[\-]*INFINITY$/) { + $f->[4] = "'$f->[4]'::$type"; + } elsif ($f->[4] =~ /AT TIME ZONE/i) { + $f->[4] = "($f->[4])"; + } + } else { + $f->[4] = "'$f->[4]'"; + } + } + } + $f->[4] = 'NULL' if ($f->[4] eq "''" && $type =~ /int|double|numeric/i); + $sql_output .= " DEFAULT $f->[4]"; + } + } + } + } + $sql_output .= ",\n"; + } + if ($self->{pkey_in_create}) { + $sql_output .= $self->_get_primary_keys($table, $self->{tables}{$table}{unique_key}); + } + $sql_output =~ s/,$//; + $sql_output .= ')'; + if (exists $self->{tables}{$table}{table_info}{on_commit}) + { + $sql_output .= ' ' . $self->{tables}{$table}{table_info}{on_commit}; + } + + if ($self->{tables}{$table}{table_info}{partitioned} && $self->{pg_supports_partition} && !$self->{disable_partition}) { + $sql_output .= " PARTITION BY " . $self->{partitions_list}{"\L$table\E"}{type} . " ("; + for (my $j = 0; $j <= $#{$self->{partitions_list}{"\L$table\E"}{columns}}; $j++) + { + $sql_output .= ', ' if ($j > 0); + $sql_output .= $self->quote_object_name($self->{partitions_list}{"\L$table\E"}{columns}[$j]); + } + $sql_output .= ")"; + } + if ( ($self->{type} ne 'FDW') && (!$self->{external_to_fdw} || (!grep(/^$table$/i, keys %{$self->{external_table}}) && !$self->{tables}{$table}{table_info}{connection})) ) { + my $withoid = _make_WITH($self->{with_oid}, $self->{tables}{$table}{table_info}); + if ($self->{use_tablespace} && $self->{tables}{$table}{table_info}{tablespace} && !grep(/^$self->{tables}{$table}{table_info}{tablespace}$/i, @{$self->{default_tablespaces}})) { + $sql_output .= " $withoid TABLESPACE $self->{tables}{$table}{table_info}{tablespace};\n"; + } else { + $sql_output .= " $withoid;\n"; + } + } elsif ( grep(/^$table$/i, keys %{$self->{external_table}}) ) { + my $program = ''; + $program = ", program '$self->{external_table}{$table}{program}'" if ($self->{external_table}{$table}{program}); + $sql_output .= " SERVER \L$self->{external_table}{$table}{directory}\E OPTIONS(filename '$self->{external_table}{$table}{directory_path}$self->{external_table}{$table}{location}', format 'csv', delimiter '$self->{external_table}{$table}{delimiter}'$program);\n"; + } elsif ($self->{is_mysql}) { + $schem = "dbname '$self->{schema}'," if ($self->{schema}); + my $r_server = $self->{fdw_server}; + my $r_table = $table; + if ($self->{tables}{$table}{table_info}{connection} =~ /([^'\/]+)\/([^']+)/) { + $r_server = $1; + $r_table = $2; + } + $sql_output .= " SERVER $r_server OPTIONS($schem table_name '$r_table');\n"; + } else { + my $tmptb = $table; + if ($self->{schema}) { + $schem = "schema '$self->{schema}',"; + } elsif ($tmptb =~ s/^([^\.]+)\.//) { + $schem = "schema '$1',"; + } + $sql_output .= " SERVER $self->{fdw_server} OPTIONS($schem table '$tmptb');\n"; + } + } + $sql_output .= $serial_sequence; + $sql_output .= $enum_str; + + # Add comments on table + if (!$self->{disable_comment} && $self->{tables}{$table}{table_info}{comment}) + { + $self->{tables}{$table}{table_info}{comment} =~ s/'/''/gs; + $sql_output .= "COMMENT ON$foreign TABLE $tbname IS E'$self->{tables}{$table}{table_info}{comment}';\n"; + } + + # Add comments on columns + if (!$self->{disable_comment}) + { + foreach my $f (sort { lc($a) cmp lc($b) } keys %{$self->{tables}{$table}{column_comments}}) + { + next unless $self->{tables}{$table}{column_comments}{$f}; + $self->{tables}{$table}{column_comments}{$f} =~ s/'/''/gs; + # Change column names + my $fname = $f; + if (exists $self->{replaced_cols}{"\L$table\E"}{lc($fname)} && $self->{replaced_cols}{"\L$table\E"}{lc($fname)}) { + $self->logit("\tReplacing column $f as " . $self->{replaced_cols}{"\L$table\E"}{lc($fname)} . "...\n", 1); + $fname = $self->{replaced_cols}{"\L$table\E"}{lc($fname)}; + } + $sql_output .= "COMMENT ON COLUMN " . $self->quote_object_name("$tbname.$fname") . " IS E'" . $self->{tables}{$table}{column_comments}{$f} . "';\n"; + } + } + + # Change ownership + if ($self->{force_owner}) + { + my $owner = $self->{tables}{$table}{table_info}{owner}; + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + $sql_output .= "ALTER$foreign $self->{tables}{$table}{table_info}{type} " . $self->quote_object_name($tbname) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + if (exists $self->{tables}{$table}{alter_index} && $self->{tables}{$table}{alter_index}) + { + foreach (@{$self->{tables}{$table}{alter_index}}) { + $sql_output .= "$_;\n"; + } + } + my $export_indexes = 1; + + if ((!$self->{tables}{$table}{table_info}{partitioned} || $self->{pg_version} >= 11 + || $self->{disable_partition}) && $self->{type} ne 'FDW') + { + # Set the indexes definition + my ($idx, $fts_idx) = $self->_create_indexes($table, 0, %{$self->{tables}{$table}{indexes}}); + $indices .= "$idx\n" if ($idx); + $fts_indices .= "$fts_idx\n" if ($fts_idx); + if (!$self->{file_per_index}) + { + $sql_output .= $indices; + $indices = ''; + $sql_output .= $fts_indices; + $fts_indices = ''; + } + + # Set the unique (and primary) key definition + $constraints .= $self->_create_unique_keys($table, $self->{tables}{$table}{unique_key}); + # Set the check constraint definition + $constraints .= $self->_create_check_constraint($table, $self->{tables}{$table}{check_constraint},$self->{tables}{$table}{field_name}, @skip_column_check); + if (!$self->{file_per_constraint}) + { + $sql_output .= $constraints; + $constraints = ''; + } + } + + if (exists $self->{tables}{$table}{alter_table} && !$self->{disable_unlogged} ) + { + $obj_type =~ s/UNLOGGED //; + foreach (@{$self->{tables}{$table}{alter_table}}) { + $sql_output .= "\nALTER $obj_type $tbname $_;\n"; + } + } + $ib++; + } + if (!$self->{quiet} && !$self->{debug}) + { + print STDERR $self->progress_bar($ib - 1, $num_total_table, 25, '=', 'tables', 'end of table export.'), "\n"; + } + + if ($sequence_output && $self->{type} ne 'FDW') + { + my $fhdl = undef; + $sequence_output = qq{ +CREATE OR REPLACE FUNCTION ora2pg_upd_autoincrement_seq (tbname text, colname text) RETURNS VOID AS \$body\$ +DECLARE + query text; + maxval bigint; + seqname text; +BEGIN + query := 'SELECT max(' || colname || ')+1 FROM ' || tbname; + EXECUTE query INTO maxval; + IF (maxval IS NOT NULL) THEN + query := \$\$SELECT (string_to_array(adsrc,''''))[2] FROM pg_attrdef WHERE adrelid = '\$\$ + || tbname || \$\$'::regclass AND adnum = (SELECT attnum FROM pg_attribute WHERE attrelid = '\$\$ + || tbname || \$\$'::regclass AND attname = '\$\$ || colname || \$\$') AND adsrc LIKE 'nextval%'\$\$; + EXECUTE query INTO seqname; + IF (seqname IS NOT NULL) THEN + query := 'ALTER SEQUENCE ' || seqname || ' RESTART WITH ' || maxval; + EXECUTE query; + END IF; + ELSE + RAISE NOTICE 'Table % is empty, you must load the AUTOINCREMENT file after data import.', tbname; + END IF; +END; +\$body\$ +LANGUAGE PLPGSQL; + +} . $sequence_output; + $sequence_output .= "DROP FUNCTION ora2pg_upd_autoincrement_seq(text, text);\n"; + $self->logit("Dumping DDL to restart autoincrement sequences into separate file : AUTOINCREMENT_$self->{output}\n", 1); + $fhdl = $self->open_export_file("AUTOINCREMENT_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $sequence_output = $self->set_search_path() . $sequence_output; + $self->dump($sql_header . $sequence_output, $fhdl); + $self->close_export_file($fhdl); + } + + if ($self->{file_per_index} && ($self->{type} ne 'FDW')) + { + my $fhdl = undef; + $self->logit("Dumping indexes to one separate file : INDEXES_$self->{output}\n", 1); + $fhdl = $self->open_export_file("INDEXES_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $indices = "-- Nothing found of type indexes\n" if (!$indices && !$self->{no_header}); + $indices =~ s/\n+/\n/gs; + $self->_restore_comments(\$indices); + $indices = $self->set_search_path() . $indices; + $self->dump($sql_header . $indices, $fhdl); + $self->close_export_file($fhdl); + $indices = ''; + if ($fts_indices) { + $fts_indices =~ s/\n+/\n/gs; + my $unaccent = ''; + if ($self->{use_lower_unaccent}) { + $unaccent = qq{ +CREATE EXTENSION IF NOT EXISTS unaccent; +CREATE OR REPLACE FUNCTION unaccent_immutable(text) +RETURNS text AS +\$\$ + SELECT lower(public.unaccent('public.unaccent', \$1)); +\$\$ LANGUAGE sql IMMUTABLE; + +}; + } elsif ($self->{use_unaccent}) { + $unaccent = qq{ +CREATE EXTENSION IF NOT EXISTS unaccent; +CREATE OR REPLACE FUNCTION unaccent_immutable(text) +RETURNS text AS +\$\$ + SELECT public.unaccent('public.unaccent', \$1); +\$\$ LANGUAGE sql IMMUTABLE; + +}; + } + # FTS TRIGGERS are exported in a separated file to be able to parallelize index creation + $self->logit("Dumping triggers for FTS indexes to one separate file : FTS_INDEXES_$self->{output}\n", 1); + $fhdl = $self->open_export_file("FTS_INDEXES_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->_restore_comments(\$fts_indices); + $fts_indices = $self->set_search_path() . $fts_indices; + $self->dump($sql_header. $unaccent . $fts_indices, $fhdl); + $self->close_export_file($fhdl); + $fts_indices = ''; + } + } + + # Dumping foreign key constraints + my $fkeys = ''; + foreach my $table (sort keys %{$self->{tables}}) + { + next if ($#{$self->{tables}{$table}{foreign_key}} < 0); + $self->logit("Dumping RI $table...\n", 1); + # Add constraint definition + if ($self->{type} ne 'FDW') { + my $create_all = $self->_create_foreign_keys($table); + if ($create_all) { + if ($self->{file_per_fkeys}) { + $fkeys .= $create_all; + } else { + if ($self->{file_per_constraint}) { + $constraints .= $create_all; + } else { + $sql_output .= $create_all; + } + } + } + } + } + + if ($self->{file_per_constraint} && ($self->{type} ne 'FDW')) + { + my $fhdl = undef; + $self->logit("Dumping constraints to one separate file : CONSTRAINTS_$self->{output}\n", 1); + $fhdl = $self->open_export_file("CONSTRAINTS_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $constraints = "-- Nothing found of type constraints\n" if (!$constraints && !$self->{no_header}); + $self->_restore_comments(\$constraints); + $self->dump($sql_header . $constraints, $fhdl); + $self->close_export_file($fhdl); + $constraints = ''; + } + + if ($fkeys) + { + my $fhdl = undef; + $self->logit("Dumping foreign keys to one separate file : FKEYS_$self->{output}\n", 1); + $fhdl = $self->open_export_file("FKEYS_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $fkeys = "-- Nothing found of type foreign keys\n" if (!$fkeys && !$self->{no_header}); + $self->_restore_comments(\$fkeys); + $fkeys = $self->set_search_path() . $fkeys; + $self->dump($sql_header . $fkeys, $fhdl); + $self->close_export_file($fhdl); + $fkeys = ''; + } + + if (!$sql_output) + { + $sql_output = "-- Nothing found of type TABLE\n" if (!$self->{no_header}); + } + else + { + $self->_restore_comments(\$sql_output); + } + + $self->dump($sql_header . $sql_output); + + # Some virtual column have been found + if ($self->{type} ne 'FDW' and scalar keys %virtual_trigger_info > 0) + { + my $trig_out = ''; + foreach my $tb (sort keys %virtual_trigger_info) { + my $tname = "virt_col_${tb}_trigger"; + $tname =~ s/\./_/g; + $tname = $self->quote_object_name($tname); + my $fname = "fct_virt_col_${tb}_trigger"; + $fname =~ s/\./_/g; + $fname = $self->quote_object_name($fname); + $trig_out .= "DROP TRIGGER $self->{pg_supports_ifexists} $tname ON " . $self->quote_object_name($tb) . " CASCADE;\n\n"; + $trig_out .= "CREATE$self->{create_or_replace} FUNCTION $fname() RETURNS trigger AS \$BODY\$\n"; + $trig_out .= "BEGIN\n"; + foreach my $c (sort keys %{$virtual_trigger_info{$tb}}) { + $trig_out .= "\tNEW.$c = $virtual_trigger_info{$tb}{$c};\n"; + } + $tb = $self->quote_object_name($tb); + $trig_out .= qq{ +RETURN NEW; +end +\$BODY\$ + LANGUAGE 'plpgsql' SECURITY DEFINER; + +CREATE TRIGGER $tname + BEFORE INSERT OR UPDATE ON $tb FOR EACH ROW + EXECUTE PROCEDURE $fname(); + +}; + } + $self->_restore_comments(\$trig_out); + if (!$self->{file_per_constraint}) { + $self->dump($trig_out); + } else { + my $fhdl = undef; + $self->logit("Dumping virtual column triggers to one separate file : VIRTUAL_COLUMNS_$self->{output}\n", 1); + $fhdl = $self->open_export_file("VIRTUAL_COLUMNS_$self->{output}"); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->dump($sql_header . $trig_out, $fhdl); + $self->close_export_file($fhdl); + } + } +} + +=head2 _get_sql_statements + +Returns a string containing the PostgreSQL compatible SQL Schema +definition. + +=cut + +sub _get_sql_statements +{ + my $self = shift; + + # Process view + if ($self->{type} eq 'VIEW') + { + $self->export_view(); + } + + # Process materialized view + elsif ($self->{type} eq 'MVIEW') + { + $self->export_mview(); + } + + # Process grant + elsif ($self->{type} eq 'GRANT') + { + $self->export_grant(); + } + + # Process sequences + elsif ($self->{type} eq 'SEQUENCE') + { + $self->export_sequence(); + } + + # Process dblink + elsif ($self->{type} eq 'DBLINK') + { + $self->export_dblink(); + } + + # Process dblink + elsif ($self->{type} eq 'DIRECTORY') + { + $self->export_directory(); + } + + # Process triggers + elsif ($self->{type} eq 'TRIGGER') + { + $self->export_trigger(); + } + + # Process queries to parallelize + elsif ($self->{type} eq 'LOAD') + { + $self->parallelize_statements(); + } + + # Process queries only + elsif ($self->{type} eq 'QUERY') + { + $self->translate_query(); + } + + # Process functions only + elsif ($self->{type} eq 'FUNCTION') + { + $self->export_function(); + } + + # Process procedures only + elsif ($self->{type} eq 'PROCEDURE') + { + $self->export_procedure(); + } + + # Process packages only + elsif ($self->{type} eq 'PACKAGE') + { + $self->export_package(); + } + + # Process types only + elsif ($self->{type} eq 'TYPE') + { + $self->export_type(); + } + + # Process TABLESPACE only + elsif ($self->{type} eq 'TABLESPACE') + { + $self->export_tablespace(); + } + + # Export as Kettle XML file + elsif ($self->{type} eq 'KETTLE') + { + $self->export_kettle(); + } + + # Process PARTITION only + elsif ($self->{type} eq 'PARTITION') + { + $self->export_partition(); + } + + # Process synonyms only + elsif ($self->{type} eq 'SYNONYM') + { + $self->export_synonym(); + } + + # Dump the database structure: tables, constraints, indexes, etc. + elsif ($self->{type} eq 'TABLE' or $self->{type} eq 'FDW') + { + $self->export_table(); + } + + # Extract data only + elsif (($self->{type} eq 'INSERT') || ($self->{type} eq 'COPY')) + { + + my $sql_output = ""; + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + + my $t0 = Benchmark->new; + + # Connect the Oracle database to gather information + if ($self->{oracle_dsn} =~ /dbi:mysql/i) { + $self->{dbh} = $self->_mysql_connection(); + } else { + $self->{dbh} = $self->_oracle_connection(); + } + + # Remove external table from data export + if (scalar keys %{$self->{external_table}} ) + { + foreach my $table (keys %{$self->{tables}}) { + if ( grep(/^$table$/i, keys %{$self->{external_table}}) ) { + delete $self->{tables}{$table}; + } + } + } + # Remove remote table from export, they must be exported using FDW export type + foreach my $table (sort keys %{$self->{tables}}) + { + if ( $self->{tables}{$table}{table_info}{connection} ) { + delete $self->{tables}{$table}; + } + } + + # Get partition information + $self->_partitions() if (!$self->{disable_partition}); + + # Ordering tables by name by default + my @ordered_tables = sort { $a cmp $b } keys %{$self->{tables}}; + if (lc($self->{data_export_order}) eq 'size') + { + @ordered_tables = sort { + ($self->{tables}{$b}{table_info}{num_rows} || $self->{tables}{$a}{table_info}{num_rows}) ? + $self->{tables}{$b}{table_info}{num_rows} <=> $self->{tables}{$a}{table_info}{num_rows} : + $a cmp $b + } keys %{$self->{tables}}; + } + + # Set SQL orders that should be in the file header + # (before the COPY or INSERT commands) + my $first_header = "$sql_header\n"; + # Add search path and constraint deferring + my $search_path = $self->set_search_path(); + if (!$self->{pg_dsn}) + { + # Set search path + if ($search_path) { + $first_header .= $self->set_search_path(); + } + # Open transaction + $first_header .= "BEGIN;\n"; + # Defer all constraints + if ($self->{defer_fkey}) { + $first_header .= "SET CONSTRAINTS ALL DEFERRED;\n\n"; + } + } + elsif (!$self->{oracle_speed}) + { + # Set search path + if ($search_path) { + $self->{dbhdest}->do($search_path) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + $self->{dbhdest}->do("BEGIN;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + + #### Defined all SQL commands that must be executed before and after data loading + my $load_file = "\n"; + foreach my $table (@ordered_tables) + { + # Remove main table partition (for MySQL "SELECT * FROM emp PARTITION (p1);" is supported from 5.6) + delete $self->{partitions}{$table} if (exists $self->{partitions}{$table} && $self->{is_mysql} && ($self->{db_version} =~ /^5\.[012345]/)); + if (-e "${dirprefix}tmp_${table}_$self->{output}") { + $self->logit("Removing incomplete export file ${dirprefix}tmp_${table}_$self->{output}\n", 1); + unlink("${dirprefix}tmp_${table}_$self->{output}"); + } + # Rename table and double-quote it if required + my $tmptb = $self->get_replaced_tbname($table); + + #### Set SQL commands that must be executed before data loading + + # Drop foreign keys if required + if ($self->{drop_fkey}) + { + $self->logit("Dropping foreign keys of table $table...\n", 1); + my @drop_all = $self->_drop_foreign_keys($table, @{$self->{tables}{$table}{foreign_key}}); + foreach my $str (@drop_all) { + chomp($str); + next if (!$str); + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $first_header .= "$str\n"; + } + } + } + + # Drop indexes if required + if ($self->{drop_indexes}) + { + $self->logit("Dropping indexes of table $table...\n", 1); + my @drop_all = $self->_drop_indexes($table, %{$self->{tables}{$table}{indexes}}) . "\n"; + foreach my $str (@drop_all) + { + chomp($str); + next if (!$str); + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $first_header .= "$str\n"; + } + } + } + + # Disable triggers of current table if requested + if ($self->{disable_triggers} && !$self->{oracle_speed}) + { + my $trig_type = 'USER'; + $trig_type = 'ALL' if (uc($self->{disable_triggers}) eq 'ALL'); + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->do("ALTER TABLE $tmptb DISABLE TRIGGER $trig_type;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $first_header .= "ALTER TABLE $tmptb DISABLE TRIGGER $trig_type;\n"; + } + } + + #### Add external data file loading if file_per_table is enable + if ($self->{file_per_table} && !$self->{pg_dsn}) + { + my $file_name = "$dirprefix${table}_$self->{output}"; + $file_name =~ s/\.(gz|bz2)$//; + $load_file .= "\\i$self->{psql_relative_path} $file_name\n"; + } + + # With partitioned table, load data direct from table partition + if (exists $self->{partitions}{$table}) + { + foreach my $pos (sort {$a <=> $b} keys %{$self->{partitions}{$table}}) + { + my $part_name = $self->{partitions}{$table}{$pos}{name}; + my $tb_name = ''; + if (!exists $self->{subpartitions}{$table}{$part_name}) { + $tb_name = $part_name; + } + $tb_name = $table . '_' . $tb_name if ($self->{prefix_partition}); + next if ($self->{allow_partition} && !grep($_ =~ /^$part_name$/i, @{$self->{allow_partition}})); + + if (exists $self->{subpartitions}{$table}{$part_name}) + { + foreach my $p (sort {$a <=> $b} keys %{$self->{subpartitions}{$table}{$part_name}}) + { + my $subpart = $self->{subpartitions}{$table}{$part_name}{$p}{name}; + next if ($self->{allow_partition} && !grep($_ =~ /^$subpart$/i, @{$self->{allow_partition}})); + my $sub_tb_name = $subpart; + $sub_tb_name =~ s/^[^\.]+\.//; # remove schema part if any + $sub_tb_name = "${tb_name}$sub_tb_name" if ($self->{prefix_partition}); + if ($self->{file_per_table} && !$self->{pg_dsn}) { + my $file_name = "$dirprefix${sub_tb_name}_$self->{output}"; + $file_name =~ s/\.(gz|bz2)$//; + $load_file .= "\\i$self->{psql_relative_path} $file_name\n"; + } + } + # Now load content of the default partion table + if ($self->{subpartitions_default}{$table}{$part_name}) + { + if (!$self->{allow_partition} || grep($_ =~ /^$self->{subpartitions_default}{$table}{$part_name}$/i, @{$self->{allow_partition}})) + { + if ($self->{file_per_table} && !$self->{pg_dsn}) + { + my $part_name = $self->{subpartitions_default}{$table}{$part_name}; + $part_name = "${tb_name}$part_name" if ($self->{prefix_partition}); + my $file_name = "$dirprefix${part_name}_$self->{output}"; + $file_name =~ s/\.(gz|bz2)$//; + $load_file .= "\\i$self->{psql_relative_path} $file_name\n"; + } + } + } + } + else + { + if ($self->{file_per_table} && !$self->{pg_dsn}) + { + my $file_name = "$dirprefix${tb_name}_$self->{output}"; + $file_name =~ s/\.(gz|bz2)$//; + $load_file .= "\\i$self->{psql_relative_path} $file_name\n"; + } + } + } + # Now load content of the default partion table + if ($self->{partitions_default}{$table}) + { + if (!$self->{allow_partition} || grep($_ =~ /^$self->{partitions_default}{$table}$/i, @{$self->{allow_partition}})) + { + if ($self->{file_per_table} && !$self->{pg_dsn}) + { + my $part_name = $self->{partitions_default}{$table}; + $part_name = $table . '_' . $part_name if ($self->{prefix_partition}); + my $file_name = "$dirprefix${part_name}_$self->{output}"; + $file_name =~ s/\.(gz|bz2)$//; + $load_file .= "\\i$self->{psql_relative_path} $file_name\n"; + } + } + } + } + + # Create temporary tables for DATADIFF + if ($self->{datadiff}) + { + my $tmptb_del = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_del_suffix}); + my $tmptb_ins = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); + my $tmptb_upd = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_upd_suffix}); + if ($self->{datadiff_work_mem}) { + $first_header .= "SET work_mem TO '" . $self->{datadiff_work_mem} . "';\n"; + } + if ($self->{datadiff_temp_buffers}) { + $first_header .= "SET temp_buffers TO '" . $self->{datadiff_temp_buffers} . "';\n"; + } + $first_header .= "LOCK TABLE $tmptb IN EXCLUSIVE MODE;\n"; + $first_header .= "CREATE TEMPORARY TABLE $tmptb_del"; + $first_header .= " (LIKE $tmptb INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES)"; + $first_header .= " ON COMMIT DROP;\n"; + $first_header .= "CREATE TEMPORARY TABLE $tmptb_ins"; + $first_header .= " (LIKE $tmptb INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES)"; + $first_header .= " ON COMMIT DROP;\n"; + $first_header .= "CREATE TEMPORARY TABLE $tmptb_upd"; + $first_header .= " (old $tmptb_del, new $tmptb_ins, changed_columns TEXT[])"; + $first_header .= " ON COMMIT DROP;\n"; + + } + + } + + if (!$self->{pg_dsn}) + { + # Write header to file + $self->dump($first_header); + + if ($self->{file_per_table}) { + # Write file loader + $self->dump($load_file); + } + } + + # Commit transaction + if ($self->{pg_dsn} && !$self->{oracle_speed}) { + my $s = $self->{dbhdest}->do("COMMIT;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + + #### + #### Proceed to data export + #### + + # Set total number of rows + $self->{global_rows} = 0; + foreach my $table (keys %{$self->{tables}}) { + if ($self->{global_where}) { + if ($self->{is_mysql} && ($self->{global_where} =~ /\s+LIMIT\s+\d+,(\d+)/)) { + $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); + } elsif ($self->{global_where} =~ /\s+ROWNUM\s+[<=>]+\s+(\d+)/) { + $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); + } + } elsif (exists $self->{where}{"\L$table\E"}) { + if ($self->{is_mysql} && ($self->{where}{"\L$table\E"} =~ /\s+LIMIT\s+\d+,(\d+)/)) { + $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); + } elsif ($self->{where}{"\L$table\E"} =~ /\s+ROWNUM\s+[<=>]+\s+(\d+)/) { + $self->{tables}{$table}{table_info}{num_rows} = $1 if ($i < $self->{tables}{$table}{table_info}{num_rows}); + } + } + $self->{global_rows} += $self->{tables}{$table}{table_info}{num_rows}; + } + + # Open a pipe for interprocess communication + my $reader = new IO::Handle; + my $writer = new IO::Handle; + + # Fork the logger process + if (!$self->{quiet} && !$self->{debug}) { + if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) || ($self->{parallel_tables} > 1)) { + $pipe = IO::Pipe->new($reader, $writer); + $writer->autoflush(1); + spawn sub { + $self->multiprocess_progressbar(); + }; + } + } + $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + + my $first_start_time = time(); + my $global_count = 0; + my $parallel_tables_count = 1; + $self->{oracle_copies} = 1 if ($self->{parallel_tables} > 1); + + # Send global startup information to pipe + if (defined $pipe) { + $pipe->writer(); + $pipe->print("GLOBAL EXPORT START TIME: $first_start_time\n"); + $pipe->print("GLOBAL EXPORT ROW NUMBER: $self->{global_rows}\n"); + } + $self->{global_start_time} = time(); + foreach my $table (@ordered_tables) + { + if ($self->{file_per_table} && !$self->{pg_dsn}) { + # Do not dump data again if the file already exists + next if ($self->file_exists("$dirprefix${table}_$self->{output}")); + } + + # Set global count + $global_count += $self->{tables}{$table}{table_info}{num_rows}; + + # Extract all column information used to determine data export. + # This hash will be used in function _howto_get_data() + %{$self->{colinfo}} = $self->_column_attributes($table, $self->{schema}, 'TABLE'); + + my $total_record = 0; + if ($self->{parallel_tables} > 1) + { + spawn sub { + $self->logit("Creating new connection to Oracle database to export table $table...\n", 1); + $self->_export_table_data($table, $dirprefix, $sql_header); + }; + $parallel_tables_count++; + + # Wait for oracle connection terminaison + while ($parallel_tables_count > $self->{parallel_tables}) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $parallel_tables_count--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + } else { + $total_record = $self->_export_table_data($table, $dirprefix, $sql_header); + } + + # Display total export position + if (!$self->{quiet} && !$self->{debug}) { + if ( ($self->{jobs} <= 1) && ($self->{oracle_copies} <= 1) && ($self->{parallel_tables} <= 1) ) { + my $last_end_time = time(); + my $dt = $last_end_time - $first_start_time; + $dt ||= 1; + my $rps = int(($total_record || $global_count) / $dt); + print STDERR $self->progress_bar(($total_record || $global_count), $self->{global_rows}, 25, '=', 'rows', "on total estimated data ($dt sec., avg: $rps recs/sec)"), "\r"; + } + } + } + if (!$self->{quiet} && !$self->{debug}) { + if ( ($self->{jobs} <= 1) && ($self->{oracle_copies} <= 1) && ($self->{parallel_tables} <= 1) ) { + print "\n"; + } + } + + # Wait for all child die + if ( ($self->{oracle_copies} > 1) || ($self->{parallel_tables} > 1) ) + { + # Wait for all child dies less the logger + my $numchild = 1; # will not wait for progressbar process + $numchild = 0 if ($self->{debug}); # in debug there is no progressbar + while (scalar keys %RUNNING_PIDS > $numchild) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + # Terminate the process logger + foreach my $k (keys %RUNNING_PIDS) { + kill(10, $k); + %RUNNING_PIDS = (); + } + # Reopen a new database handler + $self->{dbh}->disconnect() if (defined $self->{dbh}); + if ($self->{oracle_dsn} =~ /dbi:mysql/i) { + $self->{dbh} = $self->_mysql_connection(); + } else { + $self->{dbh} = $self->_oracle_connection(); + } + + } + + # Start a new transaction + if ($self->{pg_dsn} && !$self->{oracle_speed}) { + my $s = $self->{dbhdest}->do("BEGIN;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + + } + + # Remove function created to export external table + if ($self->{bfile_found} eq 'text') { + $self->logit("Removing function ora2pg_get_bfilename() used to retrieve path from BFILE.\n", 1); + my $bfile_function = "DROP FUNCTION ora2pg_get_bfilename"; + my $sth2 = $self->{dbh}->do($bfile_function); + } elsif ($self->{bfile_found} eq 'efile') { + $self->logit("Removing function ora2pg_get_efile() used to retrieve EFILE from BFILE.\n", 1); + my $efile_function = "DROP FUNCTION ora2pg_get_efile"; + my $sth2 = $self->{dbh}->do($efile_function); + } elsif ($self->{bfile_found} eq 'bytea') { + $self->logit("Removing function ora2pg_get_bfile() used to retrieve BFILE content.\n", 1); + my $efile_function = "DROP FUNCTION ora2pg_get_bfile"; + my $sth2 = $self->{dbh}->do($efile_function); + } + + #### Set SQL commands that must be executed after data loading + my $footer = ''; + my (@datadiff_tbl, @datadiff_del, @datadiff_upd, @datadiff_ins); + foreach my $table (@ordered_tables) { + + # Rename table and double-quote it if required + my $tmptb = $self->get_replaced_tbname($table); + + # DATADIFF reduction (annihilate identical deletions and insertions) and execution + if ($self->{datadiff}) { + my $tmptb_del = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_del_suffix}); + my $tmptb_upd = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_upd_suffix}); + my $tmptb_ins = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); + my @pg_colnames_nullable = @{$self->{tables}{$table}{pg_colnames_nullable}}; + my @pg_colnames_notnull = @{$self->{tables}{$table}{pg_colnames_notnull}}; + my @pg_colnames_pkey = @{$self->{tables}{$table}{pg_colnames_pkey}}; + # reduce by deleting matching (i.e. quasi "unchanged") entries from $tmptb_del and $tmptb_ins + $footer .= "WITH del AS (SELECT t, row_number() OVER (PARTITION BY t.*) rownum, ctid FROM $tmptb_del t), "; + $footer .= "ins AS (SELECT t, row_number() OVER (PARTITION BY t.*) rownum, ctid FROM $tmptb_ins t), "; + $footer .= "paired AS (SELECT del.ctid ctid1, ins.ctid ctid2 FROM del JOIN ins ON del.t IS NOT DISTINCT FROM ins.t "; + foreach my $col (@pg_colnames_nullable) { + $footer .= "AND (((del.t).$col IS NULL AND (ins.t).$col IS NULL) OR ((del.t).$col = (ins.t).$col)) "; + } + foreach my $col (@pg_colnames_notnull, @pg_colnames_pkey) { + $footer .= "AND ((del.t).$col = (ins.t).$col) "; + } + $footer .= "AND del.rownum = ins.rownum), "; + $footer .= "del_del AS (DELETE FROM $tmptb_del WHERE ctid = ANY(ARRAY(SELECT ctid1 FROM paired))), "; + $footer .= "del_ins AS (DELETE FROM $tmptb_ins WHERE ctid = ANY(ARRAY(SELECT ctid2 FROM paired))) "; + $footer .= "SELECT 1;\n"; + # convert matching delete+insert into update if configured and primary key exists + if ($self->{datadiff_update_by_pkey} && $#pg_colnames_pkey >= 0) { + $footer .= "WITH upd AS (SELECT old, new, old.ctid ctid1, new.ctid ctid2, ARRAY("; + for my $col (@pg_colnames_notnull) { + $footer .= "SELECT '$col'::TEXT WHERE old.$col <> new.$col UNION ALL "; + } + for my $col (@pg_colnames_nullable) { + $footer .= "SELECT '$col'::TEXT WHERE old.$col <> new.$col OR ((old.$col IS NULL) <> (new.$col IS NULL)) UNION ALL "; + } + $footer .= "SELECT ''::TEXT WHERE FALSE) changed_columns FROM $tmptb_del old "; + $footer .= "JOIN $tmptb_ins new USING (" . join(', ', @pg_colnames_pkey) . ")), "; + $footer .= "del_del AS (DELETE FROM $tmptb_del WHERE ctid = ANY(ARRAY(SELECT ctid1 FROM upd))), "; + $footer .= "del_ins AS (DELETE FROM $tmptb_ins WHERE ctid = ANY(ARRAY(SELECT ctid2 FROM upd))) "; + $footer .= "INSERT INTO $tmptb_upd (old, new, changed_columns) SELECT old, new, changed_columns FROM upd;\n"; + } + # call optional function specified in config to be called before actual deletion/insertion + $footer .= "SELECT " . $self->{datadiff_before} . "('" . $tmptb . "', '" . $tmptb_del . "', '" . $tmptb_upd . "', '" . $tmptb_ins . "');\n" + if ($self->{datadiff_before}); + # do actual delete + $footer .= "WITH del AS (SELECT d.delctid FROM (SELECT t, COUNT(*) c FROM $tmptb_del t GROUP BY t) s "; + $footer .= "LEFT JOIN LATERAL (SELECT ctid delctid FROM $tmptb tbl WHERE tbl IS NOT DISTINCT FROM s.t "; + foreach my $col (@pg_colnames_nullable) { + $footer .= "AND (((s.t).$col IS NULL AND tbl.$col IS NULL) OR ((s.t).$col = tbl.$col)) "; + } + foreach my $col (@pg_colnames_notnull, @pg_colnames_pkey) { + $footer .= "AND ((s.t).$col = tbl.$col) "; + } + $footer .= "LIMIT s.c) d ON TRUE) "; + $footer .= "DELETE FROM $tmptb WHERE ctid = ANY(ARRAY(SELECT delctid FROM del));\n"; + # do actual update + if ($self->{datadiff_update_by_pkey} && $#pg_colnames_pkey >= 0 && ($#pg_colnames_nullable >= 0 || $#pg_colnames_notnull >= 0)) { + $footer .= "UPDATE $tmptb SET "; + $footer .= join(', ', map { $_ . ' = (upd.new).' . $_ } @pg_colnames_notnull, @pg_colnames_nullable); + $footer .= " FROM $tmptb_upd upd WHERE "; + $footer .= join(' AND ', map { $_ . ' = (upd.old).' . $_ } @pg_colnames_pkey); + $footer .= ";\n"; + } + # do actual insert + $footer .= "INSERT INTO $tmptb SELECT * FROM $tmptb_ins;\n"; + # call optional function specified in config to be called after actual deletion/insertion + $footer .= "SELECT " . $self->{datadiff_after} . "('" . $tmptb . "', '" . $tmptb_del . "', '" . $tmptb_upd . "', '" . $tmptb_ins . "');\n" + if ($self->{datadiff_after}); + # push table names in array for bunch function call in the end + push @datadiff_tbl, $tmptb; + push @datadiff_del, $tmptb_del; + push @datadiff_upd, $tmptb_upd; + push @datadiff_ins, $tmptb_ins; + } + + + # disable triggers of current table if requested + if ($self->{disable_triggers} && !$self->{oracle_speed}) { + my $trig_type = 'USER'; + $trig_type = 'ALL' if (uc($self->{disable_triggers}) eq 'ALL'); + my $str = "ALTER TABLE $tmptb ENABLE TRIGGER $trig_type;"; + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $footer .= "$str\n"; + } + } + + # Recreate all foreign keys of the concerned tables + if ($self->{drop_fkey} && !$self->{oracle_speed}) { + my @create_all = (); + $self->logit("Restoring foreign keys of table $table...\n", 1); + push(@create_all, $self->_create_foreign_keys($table)); + foreach my $str (@create_all) { + chomp($str); + next if (!$str); + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $footer .= "$str\n"; + } + } + } + + # Recreate all indexes + if ($self->{drop_indexes} && !$self->{oracle_speed}) { + my @create_all = (); + $self->logit("Restoring indexes of table $table...\n", 1); + push(@create_all, $self->_create_indexes($table, 1, %{$self->{tables}{$table}{indexes}})); + if ($#create_all >= 0) { + foreach my $str (@create_all) { + chomp($str); + next if (!$str); + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $footer .= "$str\n"; + } + } + } + } + } + + # Insert restart sequences orders + if (($#ordered_tables >= 0) && !$self->{disable_sequence} && !$self->{oracle_speed}) { + $self->logit("Restarting sequences\n", 1); + my @restart_sequence = $self->_extract_sequence_info(); + foreach my $str (@restart_sequence) { + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->do($str) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $footer .= "$str\n"; + } + } + } + + # DATADIFF: call optional function specified in config to be called with all table names right before commit + if ($self->{datadiff} && $self->{datadiff_after_all} && $#datadiff_tbl >= 0) { + $footer .= "SELECT " . $self->{datadiff_after_all} . "(ARRAY['"; + $footer .= join("', '", @datadiff_tbl) . "'], ARRAY['"; + $footer .= join("', '", @datadiff_del) . "'], ARRAY['"; + $footer .= join("', '", @datadiff_upd) . "'], ARRAY['"; + $footer .= join("', '", @datadiff_ins) . "']);\n"; + } + + # Commit transaction + if ($self->{pg_dsn} && !$self->{oracle_speed}) { + my $s = $self->{dbhdest}->do("COMMIT;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } else { + $footer .= "COMMIT;\n\n"; + } + + # Recreate constraint an indexes if required + $self->dump("\n$footer") if (!$self->{pg_dsn} && $footer); + + my $npart = 0; + my $nsubpart = 0; + foreach my $t (sort keys %{ $self->{partitions} }) { + $npart += scalar keys %{$self->{partitions}{$t}}; + } + foreach my $t (sort keys %{ $self->{subpartitions_list} }) { + foreach my $p (sort keys %{ $self->{subpartitions_list}{$t} }) { + $nsubpart += scalar keys %{ $self->{subpartitions_list}{$t}{$p}}; + } + } + + + my $t1 = Benchmark->new; + my $td = timediff($t1, $t0); + my $timestr = timestr($td); + my $title = 'Total time to export data'; + if ($self->{ora2pg_speed}) { + $title = 'Total time to process data from Oracle'; + } elsif ($self->{oracle_speed}) { + $title = 'Total time to extract data from Oracle'; + } + $self->logit("$title from " . (scalar keys %{$self->{tables}}) . " tables ($npart partitions, $nsubpart sub-partitions) and $self->{global_rows} total rows: $timestr\n", 1); + if ($timestr =~ /^(\d+) wallclock secs/) { + my $mean = sprintf("%.2f", $self->{global_rows}/($1 || 1)); + $self->logit("Speed average: $mean rows/sec\n", 1); + } + return; + } +} + +sub fix_function_call +{ + my $self = shift; + + + $self->logit("Fixing function calls in output files...\n", 0); + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + + return unless(open(my $tfh, '<', $dirprefix . 'temp_pass2_file.dat')); + while (my $l = <$tfh>) { + chomp($l); + my ($pname, $fname, $file_name) = split(/:/, $l); + $file_to_update{$pname}{$fname} = $file_name; + } + close($tfh); + + my $child_count = 0; + # Fix call to package function in files + foreach my $pname (sort keys %file_to_update ) { + next if ($pname =~ /^ORA2PG_/); + foreach my $fname (sort keys %{ $file_to_update{$pname} } ) { + if ($self->{jobs} > 1) { + while ($child_count >= $self->{jobs}) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $child_count--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + spawn sub { + $self->requalify_package_functions($file_to_update{$pname}{$fname}); + }; + $child_count++; + } else { + $self->requalify_package_functions($file_to_update{$pname}{$fname}); + } + } + } + + # Wait for all child end + while ($child_count > 0) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $child_count--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } +} + +# Requalify function call by using double quoted if necessary and by replacing +# dot with an undescore when PACKAGE_AS_SCHEMA is disabled. +sub requalify_package_functions +{ + my ($self, $filename) = @_; + + if (open(my $fh, '<', $filename)) { + $self->set_binmode($fh); + my $content = ''; + while (<$fh>) { $content .= $_; }; + close($f); + $self->requalify_function_call(\$content); + if (open(my $fh, '>', $filename)) { + $self->set_binmode($fh); + print $fh $content; + close($fh); + } else { + print STDERR "ERROR: requalify package functions can't write to $filename, $!\n"; + return; + } + } else { + print STDERR "ERROR: requalify package functions can't read file $filename, $!\n"; + return; + } +} + +# Routine used to read input file and return content as string, +# Character / is replaces by a ; and \r are removed +sub read_input_file +{ + my ($self, $file) = @_; + + + my $content = ''; + if (open(my $fin, '<', $file)) + { + $self->set_binmode($fin) if (_is_utf8_file( $file)); + while (<$fin>) { next if /^\/$/; $content .= $_; }; + close($fin); + } else { + die "FATAL: can't read file $file, $!\n"; + } + + $content =~ s/[\r\n]\/([\r\n]|$)/;$2/gs; + $content =~ s/\r//gs; + $content =~ s/[\r\n]SHOW\s+(?:ERRORS|ERR|BTITLE|BTI|LNO|PNO|RECYCLEBIN|RECYC|RELEASE|REL|REPFOOTER|REPF|REPHEADER|REPH|SPOOL|SPOO|SGA|SQLCODE|TTITLE|TTI|USER|XQUERY|SPPARAMETERS|PARAMETERS)[^\r\n]*([\r\n]|$)/;$2/igs; + + if ($self->{is_mysql}) + { + $content =~ s/"/'/gs; + $content =~ s/`/"/gs; + } + + return $content; +} + +sub file_exists +{ + my ($self, $file) = @_; + + return 0 if ($self->{oracle_speed}); + + if ($self->{file_per_table} && !$self->{pg_dsn}) { + if (-e "$file") { + $self->logit("WARNING: Skipping dumping data to file $file, file already exists.\n", 0); + return 1; + } + } + return 0; +} + +#### +# dump table content +#### +sub _dump_table +{ + my ($self, $dirprefix, $sql_header, $table, $part_name, $is_subpart) = @_; + + my @cmd_head = (); + my @cmd_foot = (); + + # Set search path + my $search_path = $self->set_search_path(); + if (!$self->{truncate_table} && $search_path) { + push(@cmd_head,$search_path); + } + + # Rename table and double-quote it if required + my $tmptb = ''; + + # Prefix partition name with tablename, if pg_supports_partition is enabled + # direct import to partition is not allowed so import to main table. + if (!$self->{pg_supports_partition} && $part_name && $self->{prefix_partition}) { + $tmptb = $self->get_replaced_tbname($table . '_' . $part_name); + } elsif (!$self->{pg_supports_partition} && $part_name) { + $tmptb = $self->get_replaced_tbname($part_name || $table); + } else { + $tmptb = $self->get_replaced_tbname($table); + } + + # Replace Tablename by temporary table for DATADIFF (data will be inserted in real table at the end) + # !!! does not work correctly for partitions yet !!! + if ($self->{datadiff}) { + $tmptb = $self->get_tbname_with_suffix($tmptb, $self->{datadiff_ins_suffix}); + } + + # Build the header of the query + my @tt = (); + my @stt = (); + my @nn = (); + my $col_list = ''; + my $has_geometry = 0; + my $has_identity = 0; + $has_identity = 1 if (exists $self->{identity_info}{$table}); + + # Extract column information following the Oracle position order + my @fname = (); + my (@pg_colnames_nullable, @pg_colnames_notnull, @pg_colnames_pkey); + foreach my $i ( 0 .. $#{$self->{tables}{$table}{field_name}} ) + { + my $fieldname = ${$self->{tables}{$table}{field_name}}[$i]; + if (!$self->{preserve_case}) { + if (exists $self->{modify}{"\L$table\E"}) { + next if (!grep(/^\Q$fieldname\E$/i, @{$self->{modify}{"\L$table\E"}})); + } + } else { + if (exists $self->{modify}{"$table"}) { + next if (!grep(/^\Q$fieldname\E$/i, @{$self->{modify}{"$table"}})); + } + } + + my $f = $self->{tables}{"$table"}{column_info}{"$fieldname"}; + $f->[2] =~ s/\D//g; + if (!$self->{enable_blob_export} && $f->[1] =~ /blob/i) { + # user don't want to export blob + next; + } + + if (!$self->{preserve_case}) { + push(@fname, lc($fieldname)); + } else { + push(@fname, $fieldname); + } + + if ($f->[1] =~ /SDO_GEOMETRY/i) { + $self->{local_type} = $self->{type} if (!$self->{local_type}); + $has_geometry = 1; + } + + my $type = $self->_sql_type($f->[1], $f->[2], $f->[5], $f->[6], $f->[4]); + $type = "$f->[1], $f->[2]" if (!$type); + + if (uc($f->[1]) eq 'ENUM') { + $f->[1] = 'varchar'; + } + push(@stt, uc($f->[1])); + push(@tt, $type); + push(@nn, $self->{tables}{$table}{column_info}{$fieldname}); + # Change column names + my $colname = $f->[0]; + if ($self->{replaced_cols}{lc($table)}{lc($f->[0])}) { + $self->logit("\tReplacing column $f->[0] as " . $self->{replaced_cols}{lc($table)}{lc($f->[0])} . "...\n", 1); + $colname = $self->{replaced_cols}{lc($table)}{lc($f->[0])}; + } + $colname = $self->quote_object_name($colname); + if ($colname !~ /"/ && $self->is_reserved_words($colname)) { + $colname = '"' . $colname . '"'; + } + $col_list .= "$colname,"; + if ($self->is_primary_key_column($table, $fieldname)) { + push @pg_colnames_pkey, "$colname"; + } elsif ($f->[3] =~ m/^Y/) { + push @pg_colnames_nullable, "$colname"; + } else { + push @pg_colnames_notnull, "$colname"; + } + } + $col_list =~ s/,$//; + $self->{tables}{$table}{pg_colnames_nullable} = \@pg_colnames_nullable; + $self->{tables}{$table}{pg_colnames_notnull} = \@pg_colnames_notnull; + $self->{tables}{$table}{pg_colnames_pkey} = \@pg_colnames_pkey; + + my $overriding_system = ''; + if ($self->{pg_supports_identity}) { + $overriding_system = ' OVERRIDING SYSTEM VALUE' if ($has_identity); + } + + my $s_out = "INSERT INTO $tmptb ($col_list"; + if ($self->{type} eq 'COPY') { + $s_out = "\nCOPY $tmptb ($col_list"; + } + + if ($self->{type} eq 'COPY') { + $s_out .= ") FROM STDIN$self->{copy_freeze};\n"; + } else { + $s_out .= ")$overriding_system VALUES ("; + } + + # Prepare statements might work in binary mode but not WKT + # and INTERNAL because they use the call to ST_GeomFromText() + $has_geometry = 0 if ($self->{geometry_extract_type} eq 'WKB'); + + # Use prepared statement in INSERT mode and only if + # we are not exporting a row with a spatial column + my $sprep = ''; + if ($self->{pg_dsn} && !$has_geometry) { + if ($self->{type} ne 'COPY') { + $s_out .= '?,' foreach (@fname); + $s_out =~ s/,$//; + $s_out .= ")"; + $sprep = $s_out; + } + } + + # Extract all data from the current table + my $total_record = $self->ask_for_data($table, \@cmd_head, \@cmd_foot, $s_out, \@nn, \@tt, $sprep, \@stt, $part_name, $is_subpart); + + $self->{type} = $self->{local_type} if ($self->{local_type}); + $self->{local_type} = ''; + +} + +=head2 _column_comments + +This function return comments associated to columns + +=cut +sub _column_comments +{ + my ($self, $table) = @_; + + return Ora2Pg::MySQL::_column_comments($self, $table) if ($self->{is_mysql}); + + my $condition = ''; + + my $sql = "SELECT A.COLUMN_NAME,A.COMMENTS,A.TABLE_NAME,A.OWNER FROM $self->{prefix}_COL_COMMENTS A $condition"; + if ($self->{schema}) { + $sql .= "WHERE A.OWNER='$self->{schema}' "; + } else { + $sql .= " WHERE A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + $sql .= "AND A.TABLE_NAME='$table' " if ($table); + if ($self->{db_version} !~ /Release 8/) { + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + } + if (!$table) { + $sql .= $self->limit_to_objects('TABLE','TABLE_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + + my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my %data = (); + while (my $row = $sth->fetch) + { + if (!$self->{schema} && $self->{export_schema}) { + $row->[2] = "$row->[3].$row->[2]"; + } + if (!$self->{preserve_case}) { + next if (exists $self->{modify}{"\L$row->[2]\E"} && !grep(/^\Q$row->[0]\E$/i, @{$self->{modify}{"\L$row->[2]\E"}})); + } else { + next if (exists $self->{modify}{$row->[2]} && !grep(/^\Q$row->[0]\E$/i, @{$self->{modify}{$row->[2]}})); + } + $data{$row->[2]}{$row->[0]} = $row->[1]; + } + + return %data; +} + + +=head2 _create_indexes + +This function return SQL code to create indexes of a table +and triggers to create for FTS indexes. + +- $indexonly mean no FTS index output + +=cut +sub _create_indexes +{ + my ($self, $table, $indexonly, %indexes) = @_; + + my $tbsaved = $table; + # The %indexes hash can be passed from table or materialized views definition + my $objtyp = 'tables'; + if (!exists $self->{tables}{$tbsaved} && exists $self->{materialized_views}{$tbsaved}) { + $objtyp = 'materialized_views'; + } + + my %pkcollist = (); + # Save the list of column for PK to check unique index that must be removed + foreach my $consname (keys %{$self->{$objtyp}{$tbsaved}{unique_key}}) + { + next if ($self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{type} ne 'P'); + my @conscols = grep(!/^\d+$/, @{$self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{columns}}); + # save the list of column for PK to check unique index that must be removed + $pkcollist{$tbsaved} = join(", ", @conscols); + } + $pkcollist{$tbsaved} =~ s/\s+/ /g; + + $table = $self->get_replaced_tbname($table); + my @out = (); + my @fts_out = (); + # Set the index definition + foreach my $idx (sort keys %indexes) + { + # Remove cols than have only digit as name + @{$indexes{$idx}} = grep(!/^\d+$/, @{$indexes{$idx}}); + + # Cluster, bitmap join, reversed and IOT indexes will not be exported at all + # Hash indexes will be exported as btree if PG < 10 + next if ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type} =~ /JOIN|IOT|CLUSTER|REV/i); + + if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) + { + foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { + map { s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/i } @{$indexes{$idx}}; + } + } + + my @strings = (); + my $i = 0; + for (my $j = 0; $j <= $#{$indexes{$idx}}; $j++) + { + $indexes{$idx}->[$j] =~ s/''/%%ESCAPED_STRING%%/g; + while ($indexes{$idx}->[$j] =~ s/'([^']+)'/%%string$i%%/) + { + push(@strings, $1); + $i++; + } + if ($self->{plsql_pgsql}) { + $indexes{$idx}->[$j] = Ora2Pg::PLSQL::convert_plsql_code($self, $indexes{$idx}->[$j], @strings); + } + $indexes{$idx}->[$j] =~ s/%%ESCAPED_STRING%%/''/ig; + } + + # Add index opclass if required and type allow it + my %opclass_type = (); + if ($self->{use_index_opclass}) + { + my $i = 0; + for (my $j = 0; $j <= $#{$indexes{$idx}}; $j++) + { + if (exists $self->{$objtyp}{$tbsaved}{column_info}{uc($indexes{$idx}->[$j])}) + { + my $d = $self->{$objtyp}{$tbsaved}{column_info}{uc($indexes{$idx}->[$j])}; + $d->[2] =~ s/\D//g; + if ( (($self->{use_index_opclass} == 1) || ($self->{use_index_opclass} <= $d->[2])) && ($d->[1] =~ /VARCHAR/)) { + my $typ = $self->_sql_type($d->[1], $d->[2], $d->[5], $d->[6], $f->[4]); + $typ =~ s/\(.*//; + if ($typ =~ /varchar/) { + $typ = ' varchar_pattern_ops'; + } elsif ($typ =~ /text/) { + $typ = ' text_pattern_ops'; + } elsif ($typ =~ /char/) { + $typ = ' bpchar_pattern_ops'; + } + $opclass_type{$indexes{$idx}->[$j]} = "$indexes{$idx}->[$j] $typ"; + } + } + } + } + # Add parentheses to index column definition when a space is found + if (!$self->{input_file}) + { + for ($i = 0; $i <= $#{$indexes{$idx}}; $i++) + { + if ( ($indexes{$idx}->[$i] =~ /\s/) && ($indexes{$idx}->[$i] !~ /^[^\.\s]+\s+DESC$/i) ) { + $indexes{$idx}->[$i] = '(' . $indexes{$idx}->[$i] . ')'; + } + } + } + my $columns = ''; + foreach my $s (@{$indexes{$idx}}) + { + if ($s =~ /\|\|/) { + $columns .= '(' . $s . ')'; + } else { + $columns .= ((exists $opclass_type{$s}) ? $opclass_type{$s} : $s) . ", "; + } + # Add double quotes on column name if PRESERVE_CASE is enabled + foreach my $c (keys %{$self->{tables}{$tbsaved}{column_info}}) + { + $columns =~ s/\b$c\b/"$c"/ if ($self->{preserve_case} && $columns !~ /"$c"/); + } + } + $columns =~ s/, $//s; + $columns =~ s/\s+/ /gs; + my $colscompare = $columns; + $colscompare =~ s/"//g; + $colscompare =~ s/ //g; + my $columnlist = ''; + my $skip_index_creation = 0; + my %pk_hist = (); + + foreach my $consname (keys %{$self->{$objtyp}{$tbsaved}{unique_key}}) + { + my $constype = $self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{type}; + next if (($constype ne 'P') && ($constype ne 'U')); + my @conscols = grep(!/^\d+$/, @{$self->{$objtyp}{$tbsaved}{unique_key}->{$consname}{columns}}); + for ($i = 0; $i <= $#conscols; $i++) + { + # Change column names + if (exists $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}) { + $conscols[$i] = $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}; + } + } + $columnlist = join(',', @conscols); + $columnlist =~ s/"//gs; + $columnlist =~ s/\s+//gs; + if ($constype eq 'P') + { + $pk_hist{$table} = $columnlist; + } + if (lc($columnlist) eq lc($colscompare)) + { + $skip_index_creation = 1; + last; + } + } + + # Do not create the index if there already a constraint on the same column list + # or there a primary key defined on the same columns as a unique index, in both cases + # the index will be automatically created by PostgreSQL at constraint import time. + if (!$skip_index_creation) + { + my $unique = ''; + $unique = ' UNIQUE' if ($self->{$objtyp}{$tbsaved}{uniqueness}{$idx} eq 'UNIQUE'); + my $str = ''; + my $fts_str = ''; + my $concurrently = ''; + if ($self->{$objtyp}{$tbsaved}{concurrently}{$idx}) { + $concurrently = ' CONCURRENTLY'; + } + $columns = lc($columns) if (!$self->{preserve_case}); + next if ( lc($columns) eq lc($pkcollist{$tbsaved}) ); + + for ($i = 0; $i <= $#strings; $i++) { + $columns =~ s/\%\%string$i\%\%/'$strings[$i]'/; + } + + # Replace call of schema.package.function() into package.function() + $columns =~ s/\b[^\s\.]+\.([^\s\.]+\.[^\s\.]+)\s*\(/$1\(/is; + + # Do not create indexes if they are already defined as constraints + if ($self->{type} eq 'TABLE') + { + my $col_list = $columns; + $col_list =~ s/"//g; + $col_list =~ s/, /,/g; + next if (exists $pk_hist{$table} && uc($pk_hist{$table}) eq uc($col_list)); + } + + my $schm = ''; + my $idxname = ''; + if ($idx =~ /^([^\.]+)\.(.*)$/) + { + $schm = $1; + $idxname = $2; + } else { + $idxname = $idx; + } + if ($self->{indexes_renaming}) + { + if ($table =~ /^([^\.]+)\.(.*)$/) { + $schm = $1; + $idxname = $2; + } else { + $idxname = $table; + } + $idxname =~ s/"//g; + my @collist = @{$indexes{$idx}}; + # Remove double quote, DESC and parenthesys + map { s/"//g; s/.*\(([^\)]+)\).*/$1/; s/\s+DESC//i; s/::.*//; } @collist; + $idxname = $idxname . '_' . join('_', @collist); + $idxname =~ s/\s+//g; + if ($self->{indexes_suffix}) { + $idxname = substr($idxname,0,59); + } else { + $idxname = substr($idxname,0,63); + } + } + $idxname = $schm . '.' . $idxname if ($schm); + $idxname = $self->quote_object_name($idxname); + my $tb = $self->quote_object_name($table); + if ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /SPATIAL_INDEX/) + { + $str .= "CREATE INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $tb USING gist($columns)"; + } + elsif ($self->{bitmap_as_gin} && $self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} eq 'BITMAP') + { + $str .= "CREATE INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $tb USING gin($columns)"; + } + elsif ( ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /CTXCAT/) || + ($self->{context_as_trgm} && ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /FULLTEXT|CONTEXT/)) ) + { + # use pg_trgm + my @cols = split(/\s*,\s*/, $columns); + map { s/^(.*)$/unaccent_immutable($1)/; } @cols if ($self->{use_unaccent}); + $columns = join(" gin_trgm_ops, ", @cols); + $columns .= " gin_trgm_ops"; + $str .= "CREATE INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $tb USING gin($columns)"; + } + elsif (($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /FULLTEXT|CONTEXT/) && $self->{fts_index_only}) + { + my $stemmer = $self->{fts_config} || lc($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{stemmer}) || 'pg_catalog.english'; + my $dico = $stemmer; + $dico =~ s/^pg_catalog\.//; + if ($self->{use_unaccent}) { + $dico =~ s/^(..).*/$1/; + if ($fts_str !~ /CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);/s) { + $fts_str .= "CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);\n"; + $stemmer =~ s/pg_catalog\.//; + $fts_str .= "ALTER TEXT SEARCH CONFIGURATION $dico ALTER MAPPING FOR hword, hword_part, word WITH unaccent, ${stemmer}_stem;\n\n"; + } + } + # use function-based index" + my @cols = split(/\s*,\s*/, $columns); + $columns = "to_tsvector('$dico', " . join("||' '||", @cols) . ")"; + $fts_str .= "CREATE INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $tb USING gin($columns);\n"; + } + elsif (($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} =~ /FULLTEXT|CONTEXT/) && !$self->{fts_index_only}) + { + # use Full text search, then create dedicated column and trigger before the index. + map { s/"//g; } @{$indexes{$idx}}; + my $newcolname = join('_', @{$indexes{$idx}}); + $fts_str .= "\n-- Append the FTS column to the table\n"; + $fts_str .= "\nALTER TABLE $tb ADD COLUMN tsv_" . substr($newcolname,0,59) . " tsvector;\n"; + my $fctname = "tsv_${table}_" . substr($newcolname,0,59-(length($table)+1)); + my $trig_name = "trig_tsv_${table}_" . substr($newcolname,0,54-(length($table)+1)); + my $contruct_vector = ''; + my $update_vector = ''; + my $weight = 'A'; + my $stemmer = $self->{fts_config} || lc($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{stemmer}) || 'pg_catalog.english'; + my $dico = $stemmer; + $dico =~ s/^pg_catalog\.//; + if ($self->{use_unaccent}) + { + $dico =~ s/^(..).*/$1/; + if ($fts_str !~ /CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);/s) + { + $fts_str .= "CREATE TEXT SEARCH CONFIGURATION $dico (COPY = $stemmer);\n"; + $stemmer =~ s/pg_catalog\.//; + $fts_str .= "ALTER TEXT SEARCH CONFIGURATION $dico ALTER MAPPING FOR hword, hword_part, word WITH unaccent, ${stemmer}_stem;\n\n"; + } + } + if ($#{$indexes{$idx}} > 0) + { + foreach my $col (@{$indexes{$idx}}) + { + $contruct_vector .= "\t\tsetweight(to_tsvector('$dico', coalesce(new.$col,'')), '$weight') ||\n"; + $update_vector .= " setweight(to_tsvector('$dico', coalesce($col,'')), '$weight') ||"; + $weight++; + } + $contruct_vector =~ s/\|\|$/;/s; + $update_vector =~ s/\|\|$/;/s; + } + else + { + $contruct_vector = "\t\tto_tsvector('$dico', coalesce(new.$indexes{$idx}->[0],''))\n"; + $update_vector = " to_tsvector('$dico', coalesce($indexes{$idx}->[0],''))"; + } + + $fts_str .= qq{ +-- When the data migration is done without trigger, create tsvector data for all the existing records +UPDATE $tb SET tsv_$newcolname = $update_vector + +-- Trigger used to keep fts field up to date +CREATE FUNCTION $fctname() RETURNS trigger AS \$\$ +BEGIN + IF TG_OP = 'INSERT' OR new.$newcolname != old.$newcolname THEN + new.tsv_$newcolname := +$contruct_vector + END IF; + return new; +END +\$\$ LANGUAGE plpgsql; + +CREATE TRIGGER $trig_name BEFORE INSERT OR UPDATE + ON $tb + FOR EACH ROW EXECUTE PROCEDURE $fctname(); + +} if (!$indexonly); + if ($objtyp eq 'tables') + { + $str .= "CREATE$unique INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $table USING gin(tsv_$newcolname)"; + } + else + { + $fts_str .= "CREATE$unique INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $table USING gin(tsv_$newcolname)"; + } + } + elsif ($self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type} =~ /DOMAIN/i && $self->{$objtyp}{$tbsaved}{idx_type}{$idx}{type_name} !~ /SPATIAL_INDEX/) + { + $str .= "-- Was declared as DOMAIN index, please check for FTS adaptation if require\n"; + $str .= "-- CREATE$unique INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $table ($columns)"; + } + else + { + $str .= "CREATE$unique INDEX$concurrently " . $self->quote_object_name("$idxname$self->{indexes_suffix}") + . " ON $table ($columns)"; + } + if ($self->{use_tablespace} && $self->{$objtyp}{$tbsaved}{idx_tbsp}{$idx} && !grep(/^$self->{$objtyp}{$tbsaved}{idx_tbsp}{$idx}$/i, @{$self->{default_tablespaces}})) + { + $str .= " TABLESPACE $self->{$objtyp}{$tbsaved}{idx_tbsp}{$idx}"; + } + if ($str) + { + $str .= ";"; + push(@out, $str); + } + push(@fts_out, $fts_str) if ($fts_str); + } + } + + return $indexonly ? (@out,@fts_out) : (join("\n", @out), join("\n", @fts_out)); +} + +=head2 _drop_indexes + +This function return SQL code to drop indexes of a table + +=cut +sub _drop_indexes +{ + my ($self, $table, %indexes) = @_; + + my $tbsaved = $table; + $table = $self->get_replaced_tbname($table); + + my @out = (); + # Set the index definition + foreach my $idx (keys %indexes) + { + # Cluster, bitmap join, reversed and IOT indexes will not be exported at all + next if ($self->{tables}{$tbsaved}{idx_type}{$idx}{type} =~ /JOIN|IOT|CLUSTER|REV/i); + + if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) + { + foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) + { + map { s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/i } @{$indexes{$idx}}; + } + } + map { if ($_ !~ /\(.*\)/) { $_ = $self->quote_object_name($_) } } @{$indexes{$idx}}; + + my $columns = ''; + foreach my $s (@{$indexes{$idx}}) + { + if ($s =~ /\|\|/) { + $columns .= '(' . $s . ')'; + } else { + $columns .= ((exists $opclass_type{$s}) ? $opclass_type{$s} : $s) . ", "; + } + # Add double quotes on column name if PRESERVE_CASE is enabled + foreach my $c (keys %{$self->{tables}{$tbsaved}{column_info}}) + { + $columns =~ s/\b$c\b/"$c"/ if ($self->{preserve_case} && $columns !~ /"$c"/); + } + } + $columns =~ s/, $//s; + $columns =~ s/\s+//gs; + my $colscompare = $columns; + $colscompare =~ s/"//gs; + my $columnlist = ''; + my $skip_index_creation = 0; + my %pk_hist = (); + + foreach my $consname (keys %{$self->{tables}{$tbsaved}{unique_key}}) + { + my $constype = $self->{tables}{$tbsaved}{unique_key}->{$consname}{type}; + next if (($constype ne 'P') && ($constype ne 'U')); + my @conscols = grep(!/^\d+$/, @{$self->{tables}{$tbsaved}{unique_key}->{$consname}{columns}}); + for ($i = 0; $i <= $#conscols; $i++) + { + # Change column names + if (exists $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}) { + $conscols[$i] = $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}; + } + } + $columnlist = join(',', @conscols); + $columnlist =~ s/"//gs; + $columnlist =~ s/\s+//gs; + if ($constype eq 'P') + { + $pk_hist{$table} = $columnlist; + } + if (lc($columnlist) eq lc($colscompare)) { + $skip_index_creation = 1; + last; + } + } + + # Do not create the index if there already a constraint on the same column list + # the index will be automatically created by PostgreSQL at constraint import time. + if (!$skip_index_creation) + { + if ($self->{indexes_renaming}) + { + map { s/"//g; } @{$indexes{$idx}}; + $idx = $self->quote_object_name($table.'_'.join('_', @{$indexes{$idx}})); + $idx =~ s/\s+//g; + if ($self->{indexes_suffix}) { + $idx = substr($idx,0,59); + } else { + $idx = substr($idx,0,63); + } + } + if ($self->{tables}{$table}{idx_type}{$idx}{type} =~ /DOMAIN/i && $self->{tables}{$table}{idx_type}{$idx}{type_name} !~ /SPATIAL_INDEX/) + { + push(@out, "-- Declared as DOMAIN index, uncomment line below if it must be removed"); + push(@out, "-- DROP INDEX $self->{pg_supports_ifexists} \L$idx$self->{indexes_suffix}\E;"); + } else { + push(@out, "DROP INDEX $self->{pg_supports_ifexists} \L$idx$self->{indexes_suffix}\E;"); + } + } + } + + return wantarray ? @out : join("\n", @out); +} + +=head2 _exportable_indexes + +This function return the indexes that will be exported + +=cut + +sub _exportable_indexes +{ + my ($self, $table, %indexes) = @_; + + my @out = (); + # Set the index definition + foreach my $idx (keys %indexes) + { + + map { if ($_ !~ /\(.*\)/) { s/^/"/; s/$/"/; } } @{$indexes{$idx}}; + map { s/"//gs } @{$indexes{$idx}}; + my $columns = join(',', @{$indexes{$idx}}); + my $colscompare = $columns; + my $columnlist = ''; + my $skip_index_creation = 0; + foreach my $consname (keys %{$self->{tables}{$table}{unique_key}}) + { + my $constype = $self->{tables}{$table}{unique_key}->{$consname}{type}; + next if (($constype ne 'P') && ($constype ne 'U')); + my @conscols = @{$self->{tables}{$table}{unique_key}->{$consname}{columns}}; + $columnlist = join(',', @conscols); + $columnlist =~ s/"//gs; + if (lc($columnlist) eq lc($colscompare)) { + $skip_index_creation = 1; + last; + } + } + + # The index will not be created + if (!$skip_index_creation) { + push(@out, $idx); + } + } + + return @out; +} + + +=head2 is_primary_key_column + +This function return 1 when the specified column is a primary key + +=cut +sub is_primary_key_column +{ + my ($self, $table, $col) = @_; + + # Set the unique (and primary) key definition + foreach my $consname (keys %{ $self->{tables}{$table}{unique_key} }) { + next if ($self->{tables}{$table}{unique_key}->{$consname}{type} ne 'P'); + my @conscols = @{$self->{tables}{$table}{unique_key}->{$consname}{columns}}; + for (my $i = 0; $i <= $#conscols; $i++) { + if (lc($conscols[$i]) eq lc($col)) { + return 1; + } + } + } + + return 0; +} + + +=head2 _get_primary_keys + +This function return SQL code to add primary keys of a create table definition + +=cut +sub _get_primary_keys +{ + my ($self, $table, $unique_key) = @_; + + my $out = ''; + + # Set the unique (and primary) key definition + foreach my $consname (keys %$unique_key) + { + next if ($self->{pkey_in_create} && ($unique_key->{$consname}{type} ne 'P')); + my $constype = $unique_key->{$consname}{type}; + my $constgen = $unique_key->{$consname}{generated}; + my $index_name = $unique_key->{$consname}{index_name}; + my @conscols = @{$unique_key->{$consname}{columns}}; + my %constypenames = ('U' => 'UNIQUE', 'P' => 'PRIMARY KEY'); + my $constypename = $constypenames{$constype}; + for (my $i = 0; $i <= $#conscols; $i++) + { + # Change column names + if (exists $self->{replaced_cols}{"\L$table\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$table\E"}{"\L$conscols[$i]\E"}) { + $conscols[$i] = $self->{replaced_cols}{"\L$table\E"}{"\L$conscols[$i]\E"}; + } + } + map { $_ = $self->quote_object_name($_) } @conscols; + + my $columnlist = join(',', @conscols); + if ($columnlist) + { + if ($self->{pkey_in_create}) + { + if (!$self->{keep_pkey_names} || ($constgen eq 'GENERATED NAME')) { + $out .= "\tPRIMARY KEY ($columnlist)"; + } else { + $out .= "\tCONSTRAINT " . $self->quote_object_name($consname) . " PRIMARY KEY ($columnlist)"; + } + if ($self->{use_tablespace} && $self->{tables}{$table}{idx_tbsp}{$index_name} && !grep(/^$self->{tables}{$table}{idx_tbsp}{$index_name}$/i, @{$self->{default_tablespaces}})) { + $out .= " USING INDEX TABLESPACE " . $self->quote_object_name($self->{tables}{$table}{idx_tbsp}{$index_name}); + } + $out .= ",\n"; + } + } + } + $out =~ s/,$//s; + + return $out; +} + + +=head2 _create_unique_keys + +This function return SQL code to create unique and primary keys of a table + +=cut +sub _create_unique_keys +{ + my ($self, $table, $unique_key) = @_; + + my $out = ''; + + my $tbsaved = $table; + $table = $self->get_replaced_tbname($table); + + # Set the unique (and primary) key definition + foreach my $consname (keys %$unique_key) + { + next if ($self->{pkey_in_create} && ($unique_key->{$consname}{type} eq 'P')); + my $constype = $unique_key->{$consname}{type}; + my $constgen = $unique_key->{$consname}{generated}; + my $index_name = $unique_key->{$consname}{index_name}; + my $deferrable = $unique_key->{$consname}{deferrable}; + my $deferred = $unique_key->{$consname}{deferred}; + my @conscols = @{$unique_key->{$consname}{columns}}; + # Exclude unique index used in PK when column list is the same + next if (($constype eq 'U') && exists $pkcollist{$table} && ($pkcollist{$table} eq join(",", @conscols))); + + my %constypenames = ('U' => 'UNIQUE', 'P' => 'PRIMARY KEY'); + my $constypename = $constypenames{$constype}; + for (my $i = 0; $i <= $#conscols; $i++) + { + # Change column names + if (exists $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"} && $self->{replaced_cols}{"\L$tbsaved\L"}{"\L$conscols[$i]\E"}) { + $conscols[$i] = $self->{replaced_cols}{"\L$tbsaved\E"}{"\L$conscols[$i]\E"}; + } + } + # Add the partition column if it is not is the PK + if ($constype eq 'P' && exists $self->{partitions_list}{"\L$tbsaved\E"}) + { + for (my $j = 0; $j <= $#{$self->{partitions_list}{"\L$tbsaved\E"}{columns}}; $j++) + { + push(@conscols, $self->{partitions_list}{"\L$tbsaved\E"}{columns}[$j]) if (!grep(/^$self->{partitions_list}{"\L$tbsaved\E"}{columns}[$j]$/i, @conscols)); + } + } + map { $_ = $self->quote_object_name($_) } @conscols; + + my $columnlist = join(',', @conscols); + if ($columnlist) + { + if (!$self->{keep_pkey_names} || ($constgen eq 'GENERATED NAME')) { + $out .= "ALTER TABLE $table ADD $constypename ($columnlist)"; + } else { + $out .= "ALTER TABLE $table ADD CONSTRAINT \L$consname\E $constypename ($columnlist)"; + } + if ($self->{use_tablespace} && $self->{tables}{$tbsaved}{idx_tbsp}{$index_name} && !grep(/^$self->{tables}{$tbsaved}{idx_tbsp}{$index_name}$/i, @{$self->{default_tablespaces}})) { + $out .= " USING INDEX TABLESPACE $self->{tables}{$tbsaved}{idx_tbsp}{$index_name}"; + } + if ($deferrable eq "DEFERRABLE") + { + $out .= " DEFERRABLE"; + if ($deferred eq "DEFERRED") { + $out .= " INITIALLY DEFERRED"; + } + } + $out .= ";\n"; + } + } + return $out; +} + +=head2 _create_check_constraint + +This function return SQL code to create the check constraints of a table + +=cut +sub _create_check_constraint +{ + my ($self, $table, $check_constraint, $field_name, @skip_column_check) = @_; + + my $tbsaved = $table; + $table = $self->get_replaced_tbname($table); + + my $out = ''; + # Set the check constraint definition + foreach my $k (keys %{$check_constraint->{constraint}}) + { + my $chkconstraint = $check_constraint->{constraint}->{$k}{condition}; + my $validate = ''; + $validate = ' NOT VALID' if ($check_constraint->{constraint}->{$k}{validate} eq 'NOT VALIDATED'); + next if (!$chkconstraint); + my $skip_create = 0; + if (exists $check_constraint->{notnull}) + { + foreach my $col (@{$check_constraint->{notnull}}) { + $skip_create = 1, last if (lc($chkconstraint) eq lc("\"$col\" IS NOT NULL")); + } + } + if (!$skip_create) + { + if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) + { + foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) + { + $chkconstraint =~ s/"$c"/"$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}"/gsi; + $chkconstraint =~ s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/gsi; + } + } + if ($self->{plsql_pgsql}) { + $chkconstraint = Ora2Pg::PLSQL::convert_plsql_code($self, $chkconstraint); + } + foreach my $c (@$field_name) + { + # Force lower case + my $ret = $self->quote_object_name($c); + $chkconstraint =~ s/"$c"/$ret/igs; + } + $k = $self->quote_object_name($k); + + # If the column has been converted as a boolean do not export the constraint + my $converted_as_boolean = 0; + foreach my $c (@$field_name) + { + if (grep(/^$c$/i, @skip_column_check) && $chkconstraint =~ /\b$c\b/i) { + $converted_as_boolean = 1; + } + } + if (!$converted_as_boolean) + { + $chkconstraint = Ora2Pg::PLSQL::replace_oracle_function($self, $chkconstraint); + $out .= "ALTER TABLE $table ADD CONSTRAINT $k CHECK ($chkconstraint)$validate;\n"; + } + } + } + + return $out; +} + +=head2 _create_foreign_keys + +This function return SQL code to create the foreign keys of a table + +=cut +sub _create_foreign_keys +{ + my ($self, $table) = @_; + + my @out = (); + + my $tbsaved = $table; + $table = $self->get_replaced_tbname($table); + + # Add constraint definition + my @done = (); + foreach my $fkname (sort keys %{$self->{tables}{$tbsaved}{foreign_link}}) + { + next if (grep(/^$fkname$/, @done)); + + # Extract all attributes if the foreign key definition + my $state; + foreach my $h (@{$self->{tables}{$tbsaved}{foreign_key}}) + { + if (lc($h->[0]) eq lc($fkname)) + { + # @$h : CONSTRAINT_NAME,R_CONSTRAINT_NAME,SEARCH_CONDITION,DELETE_RULE,$deferrable,DEFERRED,R_OWNER,TABLE_NAME,OWNER,UPDATE_RULE,VALIDATED + push(@$state, @$h); + last; + } + } + foreach my $desttable (sort keys %{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{remote}}) + { + push(@done, $fkname); + + # This is not possible to reference a partitionned table + next if ($self->{pg_supports_partition} && exists $self->{partitions_list}{lc($desttable)}); + + # Foreign key constraint on partitionned table do not support + # NO VALID when the remote table is not partitionned + my $allow_fk_notvalid = 1; + $allow_fk_notvalid = 0 if ($self->{pg_supports_partition} && exists $self->{partitions_list}{lc($tbsaved)}); + my $str = ''; + # Add double quote to column name + map { $_ = '"' . $_ . '"' } @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{local}}; + map { $_ = '"' . $_ . '"' } @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{remote}{$desttable}}; + + # Get the name of the foreign table after replacement if any + my $subsdesttable = $self->get_replaced_tbname($desttable); + # Prefix the table name with the schema name if owner of + # remote table is not the same as local one + if ($self->{schema} && (lc($state->[6]) ne lc($state->[8]))) { + $subsdesttable = $self->quote_object_name($state->[6]) . '.' . $subsdesttable; + } + + my @lfkeys = (); + push(@lfkeys, @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{local}}); + if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) { + foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) { + map { s/"$c"/"$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}"/i } @lfkeys; + } + } + my @rfkeys = (); + push(@rfkeys, @{$self->{tables}{$tbsaved}{foreign_link}{$fkname}{remote}{$desttable}}); + if (exists $self->{replaced_cols}{"\L$desttable\E"} && $self->{replaced_cols}{"\L$desttable\E"}) + { + foreach my $c (keys %{$self->{replaced_cols}{"\L$desttable\E"}}) { + map { s/"$c"/"$self->{replaced_cols}{"\L$desttable\E"}{"\L$c\E"}"/i } @rfkeys; + } + } + for (my $i = 0; $i <= $#lfkeys; $i++) { + $lfkeys[$i] = $self->quote_object_name(split(/\s*,\s*/, $lfkeys[$i])); + } + for (my $i = 0; $i <= $#rfkeys; $i++) { + $rfkeys[$i] = $self->quote_object_name(split(/\s*,\s*/, $rfkeys[$i])); + } + $fkname = $self->quote_object_name($fkname); + $str .= "ALTER TABLE $table ADD CONSTRAINT $fkname FOREIGN KEY (" . join(',', @lfkeys) . ") REFERENCES $subsdesttable(" . join(',', @rfkeys) . ")"; + $str .= " MATCH $state->[2]" if ($state->[2]); + if ($state->[3]) { + $str .= " ON DELETE $state->[3]"; + } else { + $str .= " ON DELETE NO ACTION"; + } + if ($self->{is_mysql}) { + $str .= " ON UPDATE $state->[9]" if ($state->[9]); + } else { + if ( ($self->{fkey_add_update} eq 'ALWAYS') || ( ($self->{fkey_add_update} eq 'DELETE') && ($str =~ /ON DELETE CASCADE/) ) ) { + $str .= " ON UPDATE CASCADE"; + } + } + # if DEFER_FKEY is enabled, force constraint to be + # deferrable and defer it initially. + if (!$self->{is_mysql}) + { + $str .= (($self->{'defer_fkey'} ) ? ' DEFERRABLE' : " $state->[4]") if ($state->[4]); + $state->[5] = 'DEFERRED' if ($state->[5] =~ /^Y/); + $state->[5] ||= 'IMMEDIATE'; + $str .= " INITIALLY " . ( ($self->{'defer_fkey'} ) ? 'DEFERRED' : $state->[5] ); + if ($allow_fk_notvalid && $state->[9] eq 'NOT VALIDATED') { + $str .= " NOT VALID"; + } + } + $str .= ";\n"; + push(@out, $str); + } + } + + return wantarray ? @out : join("\n", @out); +} + +=head2 _drop_foreign_keys + +This function return SQL code to the foreign keys of a table + +=cut +sub _drop_foreign_keys +{ + my ($self, $table, @foreign_key) = @_; + + my @out = (); + + $table = $self->get_replaced_tbname($table); + + # Add constraint definition + my @done = (); + foreach my $h (@foreign_key) { + next if (grep(/^$h->[0]$/, @done)); + push(@done, $h->[0]); + my $str = ''; + $h->[0] = $self->quote_object_name($h->[0]); + $str .= "ALTER TABLE $table DROP CONSTRAINT $self->{pg_supports_ifexists} $h->[0];"; + push(@out, $str); + } + + return wantarray ? @out : join("\n", @out); +} + + +=head2 _extract_sequence_info + +This function retrieves the last value returned from the sequences in the +Oracle database. The result is a SQL script assigning the new start values +to the sequences found in the Oracle database. + +=cut +sub _extract_sequence_info +{ + my $self = shift; + + return Ora2Pg::MySQL::_extract_sequence_info($self) if ($self->{is_mysql}); + + my $sql = "SELECT DISTINCT SEQUENCE_NAME, MIN_VALUE, MAX_VALUE, INCREMENT_BY, CYCLE_FLAG, ORDER_FLAG, CACHE_SIZE, LAST_NUMBER,SEQUENCE_OWNER FROM $self->{prefix}_SEQUENCES"; + if ($self->{schema}) { + $sql .= " WHERE SEQUENCE_OWNER='$self->{schema}'"; + } else { + $sql .= " WHERE SEQUENCE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + $sql .= $self->limit_to_objects('SEQUENCE','SEQUENCE_NAME'); + + my @script = (); + + my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr ."\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + while (my $seq_info = $sth->fetchrow_hashref) { + + my $seqname = $seq_info->{SEQUENCE_NAME}; + if (!$self->{schema} && $self->{export_schema}) { + $seqname = $seq_info->{SEQUENCE_OWNER} . '.' . $seq_info->{SEQUENCE_NAME}; + } + + my $nextvalue = $seq_info->{LAST_NUMBER} + $seq_info->{INCREMENT_BY}; + my $alter = "ALTER SEQUENCE $self->{pg_supports_ifexists} " . $self->quote_object_name($seqname) . " RESTART WITH $nextvalue;"; + push(@script, $alter); + $self->logit("Extracted sequence information for sequence \"$seqname\"\n", 1); + } + $sth->finish(); + + return @script; + +} + + +=head2 _howto_get_data TABLE + +This function implements an Oracle-native data extraction. + +Returns the SQL query to use to retrieve data + +=cut + +sub _howto_get_data +{ + my ($self, $table, $name, $type, $src_type, $part_name, $is_subpart) = @_; + + # Fix a problem when the table need to be prefixed by the schema + my $realtable = $table; + $realtable =~ s/\"//g; + # Do not use double quote with mysql, but backquote + if (!$self->{is_mysql}) + { + if (!$self->{schema} && $self->{export_schema}) + { + $realtable =~ s/\./"."/; + $realtable = "\"$realtable\""; + } + else + { + $realtable = "\"$realtable\""; + my $owner = $self->{tables}{$table}{table_info}{owner} || $self->{tables}{$table}{owner} || ''; + if ($owner) + { + $owner =~ s/\"//g; + $owner = "\"$owner\""; + $realtable = "$owner.$realtable"; + } + } + } + else + { + $realtable = "\`$realtable\`"; + } + + delete $self->{nullable}{$table}; + + my $alias = 'a'; + my $str = "SELECT "; + if ($self->{tables}{$table}{table_info}{nested} eq 'YES') { + $str = "SELECT /*+ nested_table_get_refs */ "; + } + + my $extraStr = ""; + # Lookup through columns information + if ($#{$name} < 0) + { + # There a problem whe can't find any column in this table + return ''; + } + else + { + for my $k (0 .. $#{$name}) + { + my $realcolname = $name->[$k]->[0]; + my $spatial_srid = ''; + $self->{nullable}{$table}{$k} = $self->{colinfo}->{$table}{$realcolname}{nullable}; + if ($name->[$k]->[0] !~ /"/) + { + # Do not use double quote with mysql + if (!$self->{is_mysql}) { + $name->[$k]->[0] = '"' . $name->[$k]->[0] . '"'; + } else { + $name->[$k]->[0] = '`' . $name->[$k]->[0] . '`'; + } + } + if ( ( $src_type->[$k] =~ /^char/i) && ($type->[$k] =~ /(varchar|text)/i)) { + $str .= "trim($self->{trim_type} '$self->{trim_char}' FROM $name->[$k]->[0]) AS $name->[$k]->[0],"; + } elsif ($self->{is_mysql} && $src_type->[$k] =~ /bit/i) { + $str .= "BIN($name->[$k]->[0]),"; + } + # If dest type is bytea the content of the file is exported as bytea + elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /bytea/i) ) + { + $self->{bfile_found} = 'bytea'; + $str .= "ora2pg_get_bfile($name->[$k]->[0]),"; + } + # If dest type is efile the content of the file is exported to use the efile extension + elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /efile/i) ) + { + $self->{bfile_found} = 'efile'; + $str .= "ora2pg_get_efile($name->[$k]->[0]),"; + } + # Only extract path to the bfile if dest type is text. + elsif ( ($src_type->[$k] =~ /bfile/i) && ($type->[$k] =~ /text/i) ) + { + $self->{bfile_found} = 'text'; + $str .= "ora2pg_get_bfilename($name->[$k]->[0]),"; + } + elsif ( $src_type->[$k] =~ /xmltype/i) + { + if ($self->{xml_pretty}) { + $str .= "$alias.$name->[$k]->[0].extract('/').getStringVal(),"; + } else { + $str .= "$alias.$name->[$k]->[0].extract('/').getClobVal(),"; + } + } + # ArcGis Geometries + elsif ( !$self->{is_mysql} && $src_type->[$k] =~ /^(ST_|STGEOM_)/i) + { + if ($self->{geometry_extract_type} eq 'WKB') { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN SDE.ST_ASBINARY($name->[$k]->[0]) ELSE NULL END,"; + } else { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN SDE.ST_ASTEXT($name->[$k]->[0]) ELSE NULL END,"; + } + } + # Oracle geometries + elsif ( !$self->{is_mysql} && $src_type->[$k] =~ /SDO_GEOMETRY/i) + { + + # Set SQL query to get the SRID of the column + if ($self->{convert_srid} > 1) { + $spatial_srid = $self->{convert_srid}; + } else { + $spatial_srid = $self->{colinfo}->{$table}{$realcolname}{spatial_srid}; + } + + # With INSERT statement we always use WKT + if ($self->{type} eq 'INSERT') { + if ($self->{geometry_extract_type} eq 'WKB') { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN SDO_UTIL.TO_WKBGEOMETRY($name->[$k]->[0]) ELSE NULL END,"; + } elsif ($self->{geometry_extract_type} eq 'INTERNAL') { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN $name->[$k]->[0] ELSE NULL END,"; + } else { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN 'ST_GeomFromText('''||SDO_UTIL.TO_WKTGEOMETRY($name->[$k]->[0])||''','||($spatial_srid)||')' ELSE NULL END,"; + } + } else { + if ($self->{geometry_extract_type} eq 'WKB') { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN SDO_UTIL.TO_WKBGEOMETRY($name->[$k]->[0]) ELSE NULL END,"; + } elsif ($self->{geometry_extract_type} eq 'INTERNAL') { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN $name->[$k]->[0] ELSE NULL END,"; + } else { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN SDO_UTIL.TO_WKTGEOMETRY($name->[$k]->[0]) ELSE NULL END,"; + } + } + + } elsif ( $self->{is_mysql} && $src_type->[$k] =~ /geometry/i) { + + if ($self->{geometry_extract_type} eq 'WKB') { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN CONCAT('SRID=',SRID($name->[$k]->[0]),';', AsBinary($name->[$k]->[0])) ELSE NULL END,"; + } else { + $str .= "CASE WHEN $name->[$k]->[0] IS NOT NULL THEN CONCAT('SRID=',SRID($name->[$k]->[0]),';',AsText($name->[$k]->[0])) ELSE NULL END,"; + } + + } elsif ( !$self->{is_mysql} && (($src_type->[$k] =~ /clob/i) || ($src_type->[$k] =~ /blob/i)) ) { + if (!$self->{enable_blob_export} && $src_type->[$k] =~ /blob/i) { + # user don't want to export blob + next; + } + if ($self->{empty_lob_null}) { + $str .= "CASE WHEN dbms_lob.getlength($name->[$k]->[0]) = 0 THEN NULL ELSE $name->[$k]->[0] END,"; + } else { + $str .= "$name->[$k]->[0],"; + } + + } else { + + $str .= "$name->[$k]->[0],"; + + } + push(@{$self->{spatial_srid}{$table}}, $spatial_srid); + + if ($type->[$k] =~ /bytea/i && $self->{enable_blob_export}) + { + if ($self->{data_limit} >= 1000) + { + $self->{local_data_limit}{$table} = int($self->{data_limit}/10); + while ($self->{local_data_limit}{$table} > 1000) { + $self->{local_data_limit}{$table} = int($self->{local_data_limit}{$table}/10); + } + } + else + { + $self->{local_data_limit}{$table} = $self->{data_limit}; + } + $self->{local_data_limit}{$table} = $self->{blob_limit} if ($self->{blob_limit}); + } + } + $str =~ s/,$//; + } + + # If we have a BFILE that might be exported as text we need to create a function + my $bfile_function = ''; + if ($self->{bfile_found} eq 'text') { + $self->logit("Creating function ora2pg_get_bfilename( p_bfile IN BFILE ) to retrieve path from BFILE.\n", 1); + $bfile_function = qq{ +CREATE OR REPLACE FUNCTION ora2pg_get_bfilename( p_bfile IN BFILE ) RETURN +VARCHAR2 + AS + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + l_path VARCHAR2(4000); + BEGIN + IF p_bfile IS NULL + THEN RETURN NULL; + ELSE + dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); + SELECT directory_path INTO l_path FROM all_directories WHERE directory_name = l_dir; + l_dir := rtrim(l_path,'/'); + RETURN l_dir || '/' || l_fname; + END IF; + END; +}; + # If we have a BFILE that might be exported as efile we need to create a function + } elsif ($self->{bfile_found} eq 'efile') { + $self->logit("Creating function ora2pg_get_efile( p_bfile IN BFILE ) to retrieve EFILE from BFILE.\n", 1); + my $quote = ''; + $quote = "''" if ($self->{type} eq 'INSERT'); + $bfile_function = qq{ +CREATE OR REPLACE FUNCTION ora2pg_get_efile( p_bfile IN BFILE ) RETURN +VARCHAR2 + AS + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + BEGIN + IF p_bfile IS NULL THEN + RETURN NULL; + ELSE + dbms_lob.FILEGETNAME( p_bfile, l_dir, l_fname ); + RETURN '($quote' || l_dir || '$quote,$quote' || l_fname || '$quote)'; + END IF; + END; +}; + # If we have a BFILE that might be exported as bytea we need to create a + # function that exports the bfile as a binary BLOB, a HEX encoded string + } elsif ($self->{bfile_found} eq 'bytea') { + $self->logit("Creating function ora2pg_get_bfile( p_bfile IN BFILE ) to retrieve BFILE content as BLOB.\n", 1); + $bfile_function = qq{ +CREATE OR REPLACE FUNCTION ora2pg_get_bfile( p_bfile IN BFILE ) RETURN +BLOB AS + filecontent BLOB := NULL; + src_file BFILE := NULL; + l_step PLS_INTEGER := 12000; + l_dir VARCHAR2(4000); + l_fname VARCHAR2(4000); + offset NUMBER := 1; +BEGIN + IF p_bfile IS NULL THEN + RETURN NULL; + END IF; + + DBMS_LOB.FILEGETNAME( p_bfile, l_dir, l_fname ); + src_file := BFILENAME( l_dir, l_fname ); + IF src_file IS NULL THEN + RETURN NULL; + END IF; + + DBMS_LOB.FILEOPEN(src_file, DBMS_LOB.FILE_READONLY); + DBMS_LOB.CREATETEMPORARY(filecontent, true); + DBMS_LOB.LOADBLOBFROMFILE (filecontent, src_file, DBMS_LOB.LOBMAXSIZE, offset, offset); + DBMS_LOB.FILECLOSE(src_file); + RETURN filecontent; +END; +}; + } + + if ($bfile_function) + { + my $local_dbh = $self->_oracle_connection(); + my $sth2 = $local_dbh->do($bfile_function); + $local_dbh->disconnect() if ($local_dbh); + } + + # Fix empty column list with nested table + $str =~ s/ ""$/ \*/; + + if ($part_name) + { + if ($is_subpart) { + $alias = "SUBPARTITION($part_name) a"; + } else { + $alias = "PARTITION($part_name) a"; + } + } + # Force parallelism on Oracle side + if ($self->{default_parallelism_degree} > 1) + { + # Only if the number of rows is upper than PARALLEL_MIN_ROWS + $self->{tables}{$table}{table_info}{num_rows} ||= 0; + if ($self->{tables}{"\L$table\E"}{table_info}{num_rows} > $self->{parallel_min_rows}) { + $str =~ s#^SELECT #SELECT /*+ FULL(a) PARALLEL(a, $self->{default_parallelism_degree}) */ #; + } + } + $str .= " FROM $realtable $alias"; + + if (exists $self->{where}{"\L$table\E"} && $self->{where}{"\L$table\E"}) + { + ($str =~ / WHERE /) ? $str .= ' AND ' : $str .= ' WHERE '; + if (!$self->{is_mysql} || ($self->{where}{"\L$table\E"} !~ /\s+LIMIT\s+\d/)) { + $str .= '(' . $self->{where}{"\L$table\E"} . ')'; + } else { + $str .= $self->{where}{"\L$table\E"}; + } + $self->logit("\tApplying WHERE clause on table: " . $self->{where}{"\L$table\E"} . "\n", 1); + } + elsif ($self->{global_where}) + { + ($str =~ / WHERE /) ? $str .= ' AND ' : $str .= ' WHERE '; + if (!$self->{is_mysql} || ($self->{global_where} !~ /\s+LIMIT\s+\d/)) { + $str .= '(' . $self->{global_where} . ')'; + } else { + $str .= $self->{global_where}; + } + $self->logit("\tApplying WHERE global clause: " . $self->{global_where} . "\n", 1); + } + + # Automatically set the column on which query will be splitted + # to the first column with a unique key and of type NUMBER. + if ($self->{oracle_copies} > 1) + { + if (!exists $self->{defined_pk}{"\L$table\E"}) + { + foreach my $consname (keys %{$self->{tables}{$table}{unique_key}}) + { + my $constype = $self->{tables}{$table}{unique_key}->{$consname}{type}; + if (($constype eq 'P') || ($constype eq 'U')) + { + foreach my $c (@{$self->{tables}{$table}{unique_key}->{$consname}{columns}}) + { + for my $k (0 .. $#{$name}) + { + my $realcolname = $name->[$k]->[0]; + $realcolname =~ s/"//g; + if ($c eq $realcolname) + { + if ($src_type->[$k] =~ /^number\(.*,.*\)/i) + { + $self->{defined_pk}{"\L$table\E"} = "ROUND($c)"; + last; + } + elsif ($src_type->[$k] =~ /^number/i) + { + $self->{defined_pk}{"\L$table\E"} = $c; + last; + } + } + } + last if (exists $self->{defined_pk}{"\L$table\E"}); + } + } + last if (exists $self->{defined_pk}{"\L$table\E"}); + } + } + if ($self->{defined_pk}{"\L$table\E"}) + { + my $colpk = $self->{defined_pk}{"\L$table\E"}; + if ($self->{preserve_case}) { + $colpk = '"' . $colpk . '"'; + } + if ($str =~ / WHERE /) { + $str .= " AND"; + } else { + $str .= " WHERE"; + } + $str .= " ABS(MOD($colpk, $self->{oracle_copies})) = ?"; + } + } + + $self->logit("DEGUG: Query sent to Oracle: $str\n", 1); + + return $str; +} + + +=head2 _sql_type INTERNAL_TYPE LENGTH PRECISION SCALE + +This function returns the PostgreSQL data type corresponding to the +Oracle data type. + +=cut + +sub _sql_type +{ + my ($self, $type, $len, $precision, $scale, $default) = @_; + + $type = uc($type); # Force uppercase + + if ($self->{is_mysql}) { + return Ora2Pg::MySQL::_sql_type($self, $type, $len, $precision, $scale); + } + + my $data_type = ''; + + # Simplify timestamp type + $type =~ s/TIMESTAMP\(\d+\)/TIMESTAMP/; + + # Interval precision for year/month/day is not supported by PostgreSQL + if ($type =~ /INTERVAL/) { + $type =~ s/(INTERVAL\s+YEAR)\s*\(\d+\)/$1/; + $type =~ s/(INTERVAL\s+YEAR\s+TO\s+MONTH)\s*\(\d+\)/$1/; + $type =~ s/(INTERVAL\s+DAY)\s*\(\d+\)/$1/; + # maximum precision allowed for seconds is 6 + if ($type =~ /INTERVAL\s+DAY\s+TO\s+SECOND\s*\((\d+)\)/) { + if ($1 > 6) { + $type =~ s/(INTERVAL\s+DAY\s+TO\s+SECOND)\s*\(\d+\)/$1(6)/; + } + } + } + + # Overide the length + if ( ($type eq 'NUMBER') && $precision ) { + $len = $precision; + return $self->{data_type}{'NUMBER(*)'} if ($scale eq '0' && exists $self->{data_type}{'NUMBER(*)'}); + } elsif ( ($type eq 'NUMBER') && ($len == 38) ) { + if ($scale eq '0' && $precision eq '') { + # Allow custom type rewrite for NUMBER(*,0) + return $self->{data_type}{'NUMBER(*,0)'} if (exists $self->{data_type}{'NUMBER(*,0)'}); + } + $precision = $len; + } elsif ( $type =~ /CHAR/ && $len && exists $self->{data_type}{"$type($len)"}) { + return $self->{data_type}{"$type($len)"}; + } elsif ( $type =~ /RAW/ && $len && exists $self->{data_type}{"$type($len)"}) { + return $self->{data_type}{"$type($len)"}; + } elsif ( $type =~ /RAW/ && $len && $default =~ /sys_guid/i) { + return 'uuid'; + } + + if (exists $self->{data_type}{$type}) + { + if ($len) + { + if ( ($type eq "CHAR") || ($type eq "NCHAR") || ($type =~ /VARCHAR/) ) + { + # Type CHAR have default length set to 1 + # Type VARCHAR(2) must have a specified length + $len = 1 if (!$len && (($type eq "CHAR") || ($type eq "NCHAR")) ); + return "$self->{data_type}{$type}($len)"; + } + elsif ($type eq "NUMBER") + { + # This is an integer + if (!$scale) + { + if ($precision) + { + if (exists $self->{data_type}{"$type($precision)"}) { + return $self->{data_type}{"$type($precision)"}; + } + if ($self->{pg_integer_type}) + { + if ($precision < 5) { + return 'smallint'; + } elsif ($precision <= 9) { + return 'integer'; # The speediest in PG + } elsif ($precision <= 19) { + return 'bigint'; + } else { + return "numeric($precision)"; + } + } + return "numeric($precision)"; + } + elsif ($self->{pg_integer_type}) + { + # Most of the time interger should be enought? + return $self->{default_numeric} || 'bigint'; + } + } + else + { + if (exists $self->{data_type}{"$type($precision,$scale)"}) { + return $self->{data_type}{"$type($precision,$scale)"}; + } + if ($self->{pg_numeric_type}) + { + if ($precision eq '') { + return "decimal(38, $scale)"; + } elsif ($precision <= 6) { + return 'real'; + } elsif ($precision <= 15) { + return 'double precision'; + } + } + $precision = 38 if ($precision eq ''); + return "decimal($precision,$scale)"; + } + } + return "$self->{data_type}{$type}"; + } + else + { + if (($type eq 'NUMBER') && $self->{pg_integer_type}) { + return $self->{default_numeric}; + } else { + return $self->{data_type}{$type}; + } + } + } + + return $type; +} + + +=head2 _column_info TABLE OWNER + +This function implements an Oracle-native column information. + +Returns a list of array references containing the following information +elements for each column the specified table + +[( + column name, + column type, + column length, + nullable column, + default value + ... +)] + +=cut + +sub _column_info +{ + my ($self, $table, $owner, $objtype, $recurs) = @_; + + return Ora2Pg::MySQL::_column_info($self,'',$owner,'TABLE') if ($self->{is_mysql}); + + $objtype ||= 'TABLE'; + + my $condition = ''; + $condition .= "AND A.TABLE_NAME='$table' " if ($table); + if ($owner) { + $condition .= "AND A.OWNER='$owner' "; + } else { + $condition .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + if (!$table) { + $condition .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + + my $sth = ''; + if ($self->{db_version} !~ /Release 8/) { + my $exclude_mview = " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + $exclude_mview .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + $sth = $self->{dbh}->prepare(<{prefix}_TAB_COLUMNS A, ALL_OBJECTS O, ALL_TAB_COLS V +WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='$objtype' + AND A.OWNER=V.OWNER AND A.TABLE_NAME=V.TABLE_NAME AND A.COLUMN_NAME=V.COLUMN_NAME $condition + $exclude_mview +ORDER BY A.COLUMN_ID +END + if (!$sth) { + my $ret = $self->{dbh}->err; + if (!$recurs && ($ret == 942) && ($self->{prefix} eq 'DBA')) { + $self->logit("HINT: Please activate USER_GRANTS or connect using a user with DBA privilege.\n"); + $self->{prefix} = 'ALL'; + return $self->_column_info($table, $owner, $objtype, 1); + } + $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); + } + } else { + # an 8i database. + $sth = $self->{dbh}->prepare(<{prefix}_TAB_COLUMNS A, ALL_OBJECTS O +WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='$objtype' + $condition +ORDER BY A.COLUMN_ID +END + if (!$sth) { + my $ret = $self->{dbh}->err; + if (!$recurs && ($ret == 942) && ($self->{prefix} eq 'DBA')) { + $self->logit("HINT: Please activate USER_GRANTS or connect using a user with DBA privilege.\n"); + $self->{prefix} = 'ALL'; + return $self->_column_info($table, $owner, $objtype, 1); + } + $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); + } + } + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); + + # Default number of line to scan to grab the geometry type of the column. + # If it not limited, the query will scan the entire table which may take a very long time. + my $max_lines = 50000; + $max_lines = $self->{autodetect_spatial_type} if ($self->{autodetect_spatial_type} > 1); + my $spatial_gtype = 'SELECT DISTINCT c.%s.SDO_GTYPE FROM %s c WHERE ROWNUM < ' . $max_lines; + my $st_spatial_gtype = 'SELECT DISTINCT ST_GeometryType(c.%s) FROM %s c WHERE ROWNUM < ' . $max_lines; + # Set query to retrieve the SRID + my $spatial_srid = "SELECT SRID FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME=? AND COLUMN_NAME=? AND OWNER=?"; + my $st_spatial_srid = "SELECT ST_SRID(c.%s) FROM %s c"; + if ($self->{convert_srid}) { + # Translate SRID to standard EPSG SRID, may return 0 because there's lot of Oracle only SRID. + $spatial_srid = 'SELECT sdo_cs.map_oracle_srid_to_epsg(SRID) FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME=? AND COLUMN_NAME=? AND OWNER=?'; + } + # Get the dimension of the geometry by looking at the number of element in the SDO_DIM_ARRAY + my $spatial_dim = "SELECT t.SDO_DIMNAME, t.SDO_LB, t.SDO_UB FROM ALL_SDO_GEOM_METADATA m, TABLE (m.diminfo) t WHERE m.TABLE_NAME=? AND m.COLUMN_NAME=? AND OWNER=?"; + my $st_spatial_dim = "SELECT ST_DIMENSION(c.%s) FROM %s c"; + + my %data = (); + my $pos = 0; + while (my $row = $sth->fetch) + { + $row->[2] = $row->[7] if $row->[1] =~ /char/i; + + # Seems that for a NUMBER with a DATA_SCALE to 0, no DATA_PRECISION and a DATA_LENGTH of 22 + # Oracle use a NUMBER(38) instead + if ( ($row->[1] eq 'NUMBER') && ($row->[6] eq '0') && ($row->[5] eq '') && ($row->[2] == 22) ) { + $row->[2] = 38; + } + + my $tmptable = $row->[8]; + if ($self->{export_schema} && !$self->{schema}) { + $tmptable = "$row->[9].$row->[8]"; + } + + # check if this is a spatial column (srid, dim, gtype) + my @geom_inf = (); + if ($row->[1] eq 'SDO_GEOMETRY' || $row->[1] =~ /^ST_|STGEOM_/) + { + # Get the SRID of the column + if ($self->{convert_srid} > 1) { + push(@geom_inf, $self->{convert_srid}); + } + else + { + my @result = (); + $spatial_srid = $st_spatial_srid if ($row->[1] =~ /^ST_|STGEOM_/); + my $sth2 = $self->{dbh}->prepare($spatial_srid); + if (!$sth2) + { + if ($self->{dbh}->errstr !~ /ORA-01741/) { + $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); + } else { + # No SRID defined, use default one + $self->logit("WARNING: Error retreiving SRID, no matter default SRID will be used: $spatial_srid\n", 0); + } + } + else + { + if ($row->[1] =~ /^ST_|STGEOM_/) { + $sth2->execute($row->[0]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } else { + $sth2->execute($row->[8],$row->[0],$row->[9]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + while (my $r = $sth2->fetch) { + push(@result, $r->[0]) if ($r->[0] =~ /\d+/); + } + $sth2->finish(); + } + if ($#result == 0) { + push(@geom_inf, $result[0]); + } elsif ($self->{default_srid}) { + push(@geom_inf, $self->{default_srid}); + } else { + push(@geom_inf, 0); + } + } + + # Grab constraint type and dimensions from index definition + my $found_contraint = 0; + foreach my $idx (keys %{$self->{tables}{$tmptable}{idx_type}}) { + if (exists $self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}) { + foreach my $c (@{$self->{tables}{$tmptable}{indexes}{$idx}}) { + if ($c eq $row->[0]) { + if ($self->{tables}{$tmptable}{idx_type}{$idx}{type_dims}) { + $found_dims = $self->{tables}{$tmptable}{idx_type}{$idx}{type_dims}; + } + if ($self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}) { + $found_contraint = $GTYPE{$self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}} || $self->{tables}{$tmptable}{idx_type}{$idx}{type_constraint}; + } + } + } + } + } + + # Get the dimension of the geometry column + if (!$found_dims) + { + $spatial_dim = $st_spatial_dim if ($row->[1] =~ /^ST_|STGEOM_/); + $sth2 = $self->{dbh}->prepare($spatial_dim); + if (!$sth2) { + $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); + } + if ($row->[1] =~ /^ST_|STGEOM_/) { + $sth2->execute($row->[0]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } else { + $sth2->execute($row->[8],$row->[0],$row->[9]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + my $count = 0; + while (my $r = $sth2->fetch) { + $count++; + } + $sth2->finish(); + push(@geom_inf, $count); + } else { + push(@geom_inf, $found_dims); + } + + # Set dimension and type of the spatial column + if (!$found_contraint && $self->{autodetect_spatial_type}) + { + # Get spatial information + my $colname = $row->[9] . "." . $row->[8]; + my $squery = sprintf($spatial_gtype, $row->[0], $colname); + if ($row->[1] =~ /^ST_|STGEOM_/) { + $squery = sprintf($st_spatial_gtype, $row->[0], $colname); + } + my $sth2 = $self->{dbh}->prepare($squery); + if (!$sth2) { + $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); + } + $sth2->execute or $self->logit("FATAL: _column_info() " . $self->{dbh}->errstr . "\n", 0, 1); + my @result = (); + while (my $r = $sth2->fetch) + { + if ($r->[0] =~ /(\d)$/) { + push(@result, $ORA2PG_SDO_GTYPE{$1}); + } elsif ($r->[0] =~ /ST_(.*)$/) { + push(@result, $1); + } + } + $sth2->finish(); + if ($#result == 0) { + push(@geom_inf, $result[0]); + } else { + push(@geom_inf, join(',', @result)); + } + } elsif ($found_contraint) { + push(@geom_inf, $found_contraint); + + } else { + push(@geom_inf, $ORA2PG_SDO_GTYPE{0}); + } + } + + if (!$self->{schema} && $self->{export_schema}) + { + next if (exists $self->{modify}{"\L$tmptable\E"} && !grep(/^\Q$row->[0]\E$/i, @{$self->{modify}{"\L$tmptable\E"}})); + push(@{$data{$tmptable}{"$row->[0]"}}, (@$row, $pos, @geom_inf)); + } + else + + { + if (!$self->{preserve_case}) { + next if (exists $self->{modify}{"\L$row->[8]\E"} && !grep(/^\Q$row->[0]\E$/i, @{$self->{modify}{"\L$row->[8]\E"}})); + } else { + next if (exists $self->{modify}{$row->[8]} && !grep(/^\Q$row->[0]\E$/i, @{$self->{modify}{$row->[8]}})); + } + push(@{$data{"$row->[8]"}{"$row->[0]"}}, (@$row, $pos, @geom_inf)); + } + + $pos++; + } + + return %data; +} + +sub _column_attributes +{ + my ($self, $table, $owner, $objtype) = @_; + + return Ora2Pg::MySQL::_column_attributes($self,'',$owner,'TABLE') if ($self->{is_mysql}); + + $objtype ||= 'TABLE'; + + my $condition = ''; + $condition .= "AND A.TABLE_NAME='$table' " if ($table); + if ($owner) { + $condition .= "AND A.OWNER='$owner' "; + } else { + $condition .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + if (!$table) { + $condition .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + + my $sth = ''; + if ($self->{db_version} !~ /Release 8/) { + $sth = $self->{dbh}->prepare(<{prefix}_TAB_COLUMNS A, ALL_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='$objtype' $condition +ORDER BY A.COLUMN_ID +END + if (!$sth) { + $self->logit("FATAL: _column_attributes() " . $self->{dbh}->errstr . "\n", 0, 1); + } + } else { + # an 8i database. + $sth = $self->{dbh}->prepare(<{prefix}_TAB_COLUMNS A, ALL_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='$objtype' $condition +ORDER BY A.COLUMN_ID +END + if (!$sth) { + $self->logit("FATAL: _column_attributes() " . $self->{dbh}->errstr . "\n", 0, 1); + } + } + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: _column_attributes() " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + if ($self->{export_schema} && !$self->{schema}) { + $data{"$row->[4].$row->[3]"}{"$row->[0]"}{nullable} = $row->[1]; + $data{"$row->[4].$row->[3]"}{"$row->[0]"}{default} = $row->[2]; + } else { + $data{$row->[3]}{"$row->[0]"}{nullable} = $row->[1]; + $data{$row->[3]}{"$row->[0]"}{default} = $row->[2]; + } + my $f = $self->{tables}{"$table"}{column_info}{"$row->[0]"}; + if ( ($f->[1] =~ /SDO_GEOMETRY/i) && ($self->{convert_srid} <= 1) ) { + $spatial_srid = "SELECT COALESCE(SRID, $self->{default_srid}) FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME='\U$table\E' AND COLUMN_NAME='$row->[0]' AND OWNER='\U$self->{tables}{$table}{table_info}{owner}\E'"; + if ($self->{convert_srid} == 1) { + $spatial_srid = "SELECT COALESCE(sdo_cs.map_oracle_srid_to_epsg(SRID), $self->{default_srid}) FROM ALL_SDO_GEOM_METADATA WHERE TABLE_NAME='\U$table\E' AND COLUMN_NAME='$row->[0]' AND OWNER='\U$self->{tables}{$table}{table_info}{owner}\E'"; + } + my $sth2 = $self->{dbh}->prepare($spatial_srid); + if (!$sth2) { + if ($self->{dbh}->errstr !~ /ORA-01741/) { + $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } else { + # No SRID defined, use default one + $spatial_srid = $self->{default_srid} || '0'; + $self->logit("WARNING: Error retreiving SRID, no matter default SRID will be used: $spatial_srid\n", 0); + } + } else { + $sth2->execute() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my @result = (); + while (my $r = $sth2->fetch) { + push(@result, $r->[0]) if ($r->[0] =~ /\d+/); + } + $sth2->finish(); + if ($self->{export_schema} && !$self->{schema}) { + $data{"$row->[4].$row->[3]"}{"$row->[0]"}{spatial_srid} = $result[0] || $self->{default_srid} || '0'; + } else { + $data{$row->[3]}{"$row->[0]"}{spatial_srid} = $result[0] || $self->{default_srid} || '0'; + } + } + } + } + + return %data; +} + +sub _encrypted_columns +{ + my ($self, $table, $owner) = @_; + + return Ora2Pg::MySQL::_encrypted_columns($self,'',$owner) if ($self->{is_mysql}); + + # Encryption appears in version 10 only + return if ($self->{db_version} =~ /Release [8|9]/); + + my $condition = ''; + $condition .= "AND A.TABLE_NAME='$table' " if ($table); + if ($owner) { + $condition .= "AND A.OWNER='$owner' "; + } else { + $condition .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + if (!$table) { + $condition .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + $condition =~ s/^\s*AND /WHERE /s; + + my $sth = $self->{dbh}->prepare(<{prefix}_ENCRYPTED_COLUMNS A +$condition +END + if (!$sth) { + $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + if ($self->{export_schema} && !$self->{schema}) { + $data{"$row->[2].$row->[1].$row->[0]"} = $row->[3]; + } else { + $data{"$row->[1].$row->[0]"} = $row->[3]; + } + } + + return %data; +} + + + +=head2 _unique_key TABLE OWNER + +This function implements an Oracle-native unique (including primary) +key column information. + +Returns a hash of hashes in the following form: + ( owner => table => constraintname => (type => 'PRIMARY', + columns => ('a', 'b', 'c')), + owner => table => constraintname => (type => 'UNIQUE', + columns => ('b', 'c', 'd')), + etc. + ) + +=cut + +sub _unique_key +{ + my ($self, $table, $owner, $type) = @_; + + return Ora2Pg::MySQL::_unique_key($self,$table,$owner) if ($self->{is_mysql}); + + my %result = (); + + my @accepted_constraint_types = (); + if ($type) { + push @accepted_constraint_types, "'$type'"; + } else { + push @accepted_constraint_types, "'P'" unless($self->{skip_pkeys}); + push @accepted_constraint_types, "'U'" unless($self->{skip_ukeys}); + } + return %result unless(@accepted_constraint_types); + + my $cons_types = '('. join(',', @accepted_constraint_types) .')'; + + my $indexname = "'' AS INDEX_NAME"; + if ($self->{db_version} !~ /Release 8/) { + $indexname = 'B.INDEX_NAME'; + } + # Get columns of all the table in the specified schema or excluding the list of system schema + my $sql = qq{SELECT DISTINCT A.COLUMN_NAME,A.CONSTRAINT_NAME,A.OWNER,A.POSITION,B.CONSTRAINT_NAME,B.CONSTRAINT_TYPE,B.DEFERRABLE,B.DEFERRED,B.GENERATED,B.TABLE_NAME,B.OWNER,$indexname +FROM $self->{prefix}_CONS_COLUMNS A JOIN $self->{prefix}_CONSTRAINTS B ON (B.CONSTRAINT_NAME = A.CONSTRAINT_NAME AND B.OWNER = A.OWNER) +}; + if ($owner) { + $sql .= " WHERE A.OWNER = '$owner'"; + } else { + $sql .= " WHERE A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + $sql .= " AND B.CONSTRAINT_TYPE IN $cons_types"; + $sql .= " AND B.TABLE_NAME='$table'" if ($table); + $sql .= " AND B.STATUS='ENABLED' "; + if ($self->{db_version} !~ /Release 8/) { + $sql .= " AND (B.OWNER, B.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + $sql .= " AND (B.OWNER, B.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + } + + # Get the list of constraints in the specified schema or excluding the list of system schema + my @tmpparams = (); + if ($self->{type} ne 'SHOW_REPORT') + { + $sql .= $self->limit_to_objects('UKEY|TABLE', 'B.CONSTRAINT_NAME|B.TABLE_NAME'); + push(@tmpparams, @{$self->{query_bind_params}}); + $sql .= $self->limit_to_objects('UKEY', 'B.CONSTRAINT_NAME'); + push(@tmpparams, @{$self->{query_bind_params}}); + } + $sql .= " ORDER BY A.POSITION"; + + my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@tmpparams) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); + + while (my $row = $sth->fetch) + { + my $name = $row->[9]; + if (!$self->{schema} && $self->{export_schema}) + { + $name = "$row->[10].$row->[9]"; + } + if (!exists $result{$name}{$row->[4]}) + { + $result{$name}{$row->[4]} = { (type => $row->[5], 'generated' => $row->[8], 'index_name' => $row->[11], 'deferrable' => $row->[6], 'deferred' => $row->[7], columns => ()) }; + push(@{ $result{$name}{$row->[4]}->{columns} }, $row->[0]) if ($row->[4] !~ /^SYS_NC/i); + } + elsif ($row->[4] !~ /^SYS_NC/i) + { + push(@{ $result{$name}{$row->[4]}->{columns} }, $row->[0]); + } + } + return %result; +} + +=head2 _check_constraint TABLE OWNER + +This function implements an Oracle-native check constraint +information. + +Returns a hash of lists of all column names defined as check constraints +for the specified table and constraint name. + +=cut + +sub _check_constraint +{ + my($self, $table, $owner) = @_; + + my $condition = ''; + $condition .= "AND TABLE_NAME='$table' " if ($table); + if ($owner) { + $condition .= "AND OWNER = '$owner' "; + } else { + $condition .= "AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + $condition .= $self->limit_to_objects('CKEY|TABLE', 'CONSTRAINT_NAME|TABLE_NAME'); + + my $sql = qq{ +SELECT A.CONSTRAINT_NAME,A.R_CONSTRAINT_NAME,A.SEARCH_CONDITION,A.DELETE_RULE,A.DEFERRABLE,A.DEFERRED,A.R_OWNER,A.TABLE_NAME,A.OWNER,A.VALIDATED +FROM $self->{prefix}_CONSTRAINTS A +WHERE A.CONSTRAINT_TYPE='C' $condition +AND A.STATUS='ENABLED' +}; + + if ($self->{db_version} !~ /Release 8/) { + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + } + my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + if ($self->{export_schema} && !$self->{schema}) { + $row->[7] = "$row->[8].$row->[7]"; + } + $data{$row->[7]}{constraint}{$row->[0]}{condition} = $row->[2]; + $data{$row->[7]}{constraint}{$row->[0]}{validate} = $row->[9]; + } + + return %data; +} + +=head2 _foreign_key TABLE OWNER + +This function implements an Oracle-native foreign key reference +information. + +Returns a list of hash of hash of array references. Ouf! Nothing very difficult. +The first hash is composed of all foreign key names. The second hash has just +two keys known as 'local' and 'remote' corresponding to the local table where +the foreign key is defined and the remote table referenced by the key. + +The foreign key name is composed as follows: + + 'local_table_name->remote_table_name' + +Foreign key data consists in two arrays representing at the same index for the +local field and the remote field where the first one refers to the second one. +Just like this: + + @{$link{$fkey_name}{local}} = @local_columns; + @{$link{$fkey_name}{remote}} = @remote_columns; + +=cut + +sub _foreign_key +{ + my ($self, $table, $owner) = @_; + + return Ora2Pg::MySQL::_foreign_key($self,$table,$owner) if ($self->{is_mysql}); + + my @tmpparams = (); + my $condition = ''; + $condition .= "AND CONS.TABLE_NAME='$table' " if ($table); + if ($owner) { + $condition .= "AND CONS.OWNER = '$owner' "; + } else { + $condition .= "AND CONS.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + $condition .= $self->limit_to_objects('FKEY|TABLE','CONS.CONSTRAINT_NAME|CONS.TABLE_NAME'); + + my $deferrable = $self->{fkey_deferrable} ? "'DEFERRABLE' AS DEFERRABLE" : "DEFERRABLE"; + my $defer = $self->{fkey_deferrable} ? "'DEFERRABLE' AS DEFERRABLE" : "CONS.DEFERRABLE"; + + my $sql = <{prefix}_CONSTRAINTS CONS + LEFT JOIN $self->{prefix}_CONS_COLUMNS COLS ON (COLS.CONSTRAINT_NAME = CONS.CONSTRAINT_NAME AND COLS.OWNER = CONS.OWNER AND COLS.TABLE_NAME = CONS.TABLE_NAME) + LEFT JOIN $self->{prefix}_CONSTRAINTS CONS_R ON (CONS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND CONS_R.OWNER = CONS.R_OWNER) + LEFT JOIN $self->{prefix}_CONS_COLUMNS COLS_R ON (COLS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND COLS_R.POSITION=COLS.POSITION AND COLS_R.OWNER = CONS.R_OWNER) +WHERE CONS.CONSTRAINT_TYPE = 'R' $condition +END + $sql .= "\nAND (CONS.OWNER, CONS.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + $sql .= " AND (CONS.OWNER, CONS.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + + $sql .= "\nORDER BY CONS.TABLE_NAME, CONS.CONSTRAINT_NAME, COLS.POSITION"; + + if ($self->{db_version} =~ /Release 8/) { + $sql = <{prefix}_CONSTRAINTS CONS, $self->{prefix}_CONS_COLUMNS COLS, $self->{prefix}_CONSTRAINTS CONS_R, $self->{prefix}_CONS_COLUMNS COLS_R +WHERE CONS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND CONS_R.OWNER = CONS.R_OWNER + AND COLS.CONSTRAINT_NAME = CONS.CONSTRAINT_NAME AND COLS.OWNER = CONS.OWNER AND COLS.TABLE_NAME = CONS.TABLE_NAME + AND COLS_R.CONSTRAINT_NAME = CONS.R_CONSTRAINT_NAME AND COLS_R.POSITION=COLS.POSITION AND COLS_R.OWNER = CONS.R_OWNER + AND CONS.CONSTRAINT_TYPE = 'R' $condition +ORDER BY CONS.TABLE_NAME, CONS.CONSTRAINT_NAME, COLS.POSITION +END + } + my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); + + my %data = (); + my %link = (); + #my @tab_done = (); + while (my $row = $sth->fetch) { + my $local_table = $row->[0]; + my $remote_table = $row->[3]; + if (!$self->{schema} && $self->{export_schema}) { + $local_table = "$row->[10].$row->[0]"; + $remote_table = "$row->[11].$row->[3]"; + } + if (!$self->{preserve_case}) { + next if (exists $self->{modify}{"\L$local_table\E"} && !grep(/^\Q$row->[2]\E$/i, @{$self->{modify}{"\L$local_table\E"}})); + next if (exists $self->{modify}{"\L$remote_table\E"} && !grep(/^\Q$row->[5]\E$/i, @{$self->{modify}{"\L$remote_table\E"}})); + } else { + next if (exists $self->{modify}{$local_table} && !grep(/^\Q$row->[2]\E$/i, @{$self->{modify}{$local_table}})); + next if (exists $self->{modify}{$remote_table} && !grep(/^\Q$row->[5]\E$/i, @{$self->{modify}{$remote_table}})); + } + push(@{$data{$local_table}}, [ ($row->[1],$row->[4],$row->[6],$row->[7],$row->[8],$row->[9],$row->[11],$row->[0],$row->[10],$row->[14]) ]); + # TABLENAME CONSTNAME COLNAME + push(@{$link{$local_table}{$row->[1]}{local}}, $row->[2]); + # TABLENAME CONSTNAME TABLENAME COLNAME + push(@{$link{$local_table}{$row->[1]}{remote}{$remote_table}}, $row->[5]); + } + + return \%link, \%data; +} + + +=head2 _get_privilege + +This function implements an Oracle-native object priviledge information. + +Returns a hash of all priviledge. + +=cut + +sub _get_privilege +{ + my($self) = @_; + + # If the user is given as not DBA, do not look at tablespace + if ($self->{user_grants}) { + $self->logit("WARNING: Exporting privilege as non DBA user is not allowed, see USER_GRANT\n", 0); + return; + } + + return Ora2Pg::MySQL::_get_privilege($self) if ($self->{is_mysql}); + + my %privs = (); + my %roles = (); + + # Retrieve all privilege per table defined in this database + my $str = "SELECT b.GRANTEE,b.OWNER,b.TABLE_NAME,b.PRIVILEGE,a.OBJECT_TYPE,b.GRANTABLE FROM DBA_TAB_PRIVS b, DBA_OBJECTS a"; + if ($self->{schema}) { + $str .= " WHERE b.GRANTOR = '$self->{schema}'"; + } else { + $str .= " WHERE b.GRANTOR NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + $str .= " AND b.TABLE_NAME=a.OBJECT_NAME AND a.OWNER=b.GRANTOR"; + if ($self->{grant_object} && $self->{grant_object} ne 'USER') { + $str .= " AND a.OBJECT_TYPE = '\U$self->{grant_object}\E'"; + } else { + $str .= " AND a.OBJECT_TYPE <> 'TYPE'"; + } + $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'b.GRANTEE|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME'); + + if (!$self->{export_invalid}) { + $str .= " AND a.STATUS='VALID'"; + } + #$str .= " ORDER BY b.TABLE_NAME, b.GRANTEE"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + next if ($row->[0] eq 'PUBLIC'); + if (!$self->{schema} && $self->{export_schema}) { + $row->[2] = "$row->[1].$row->[2]"; + } + $privs{$row->[2]}{type} = $row->[4]; + $privs{$row->[2]}{owner} = $row->[1] if (!$privs{$row->[2]}{owner}); + if ($row->[5] eq 'YES') { + $privs{$row->[2]}{grantable} = $row->[5]; + } + push(@{$privs{$row->[2]}{privilege}{$row->[0]}}, $row->[3]); + push(@{$roles{owner}}, $row->[1]) if (!grep(/^$row->[1]$/, @{$roles{owner}})); + push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); + } + $sth->finish(); + + # Retrieve all privilege per column table defined in this database + $str = "SELECT b.GRANTEE,b.OWNER,b.TABLE_NAME,b.PRIVILEGE,b.COLUMN_NAME FROM DBA_COL_PRIVS b, DBA_OBJECTS a"; + if ($self->{schema}) { + $str .= " WHERE b.GRANTOR = '$self->{schema}'"; + } else { + $str .= " WHERE b.GRANTOR NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + if (!$self->{export_invalid}) { + $str .= " AND a.STATUS='VALID'"; + } + $str .= " AND b.TABLE_NAME=a.OBJECT_NAME AND a.OWNER=b.GRANTOR AND a.OBJECT_TYPE <> 'TYPE'"; + if ($self->{grant_object} && $self->{grant_object} ne 'USER') { + $str .= " AND a.OBJECT_TYPE = '\U$self->{grant_object}\E'"; + } else { + $str .= " AND a.OBJECT_TYPE <> 'TYPE'"; + } + $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'b.GRANTEE|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME|b.TABLE_NAME'); + + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[2] = "$row->[1].$row->[2]"; + } + $privs{$row->[2]}{owner} = $row->[1] if (!$privs{$row->[2]}{owner}); + push(@{$privs{$row->[2]}{column}{$row->[4]}{$row->[0]}}, $row->[3]); + push(@{$roles{owner}}, $row->[1]) if (!grep(/^$row->[1]$/, @{$roles{owner}})); + push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); + } + $sth->finish(); + + # Search if users have admin rights + my @done = (); + foreach my $r (@{$roles{owner}}, @{$roles{grantee}}) { + next if (grep(/^$r$/, @done)); + push(@done, $r); + # Get all system priviledge given to a role + $str = "SELECT PRIVILEGE,ADMIN_OPTION FROM DBA_SYS_PRIVS WHERE GRANTEE = '$r'"; + $str .= " " . $self->limit_to_objects('GRANT', 'GRANTEE'); + #$str .= " ORDER BY PRIVILEGE"; + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + push(@{$roles{admin}{$r}{privilege}}, $row->[0]); + push(@{$roles{admin}{$r}{admin_option}}, $row->[1]); + } + $sth->finish(); + } + # Now try to find if it's a user or a role + foreach my $u (@done) { + $str = "SELECT GRANTED_ROLE FROM DBA_ROLE_PRIVS WHERE GRANTEE = '$u'"; + $str .= " " . $self->limit_to_objects('GRANT', 'GRANTEE'); + #$str .= " ORDER BY GRANTED_ROLE"; + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + push(@{$roles{role}{$u}}, $row->[0]); + } + $str = "SELECT USERNAME FROM DBA_USERS WHERE USERNAME = '$u'"; + $str .= " " . $self->limit_to_objects('GRANT', 'USERNAME'); + #$str .= " ORDER BY USERNAME"; + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + $roles{type}{$u} = 'USER'; + } + next if $roles{type}{$u}; + $str = "SELECT ROLE,PASSWORD_REQUIRED FROM DBA_ROLES WHERE ROLE='$u'"; + $str .= " " . $self->limit_to_objects('GRANT', 'ROLE'); + #$str .= " ORDER BY ROLE"; + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + $roles{type}{$u} = 'ROLE'; + $roles{password_required}{$u} = $row->[1]; + } + $sth->finish(); + } + + return (\%privs, \%roles); +} + +=head2 _get_security_definer + +This function implements an Oracle-native functions security definer / current_user information. + +Returns a hash of all object_type/function/security. + +=cut + +sub _get_security_definer +{ + my ($self, $type) = @_; + + return Ora2Pg::MySQL::_get_security_definer($self, $type) if ($self->{is_mysql}); + + my %security = (); + + # This table does not exists before 10g + return if ($self->{db_version} =~ /Release [89]/); + + # Retrieve security privilege per function defined in this database + # Version of Oracle 10 does not have the OBJECT_TYPE column. + my $str = "SELECT AUTHID,OBJECT_TYPE,OBJECT_NAME,OWNER FROM $self->{prefix}_PROCEDURES"; + if ($self->{db_version} =~ /Release 10/) { + $str = "SELECT AUTHID,'ALL' AS OBJECT_TYPE,OBJECT_NAME,OWNER FROM $self->{prefix}_PROCEDURES"; + } + if ($self->{schema}) { + $str .= " WHERE OWNER = '$self->{schema}'"; + } else { + $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + if ( $type && ($self->{db_version} !~ /Release 10/) ) { + $str .= " AND OBJECT_TYPE='$type'"; + } + $str .= " " . $self->limit_to_objects('FUNCTION|PROCEDURE|PACKAGE|TRIGGER', 'OBJECT_NAME|OBJECT_NAME|OBJECT_NAME|OBJECT_NAME'); + #$str .= " ORDER BY OBJECT_NAME"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + next if (!$row->[0]); + if (!$self->{schema} && $self->{export_schema}) { + $row->[2] = "$row->[3].$row->[2]"; + } + $security{$row->[2]}{security} = $row->[0]; + $security{$row->[2]}{owner} = $row->[3]; + } + $sth->finish(); + + return (\%security); +} + + + + +=head2 _get_indexes TABLE OWNER + +This function implements an Oracle-native indexes information. + +Returns a hash of an array containing all unique indexes and a hash of +array of all indexe names which are not primary keys for the specified table. + +=cut + +sub _get_indexes +{ + my ($self, $table, $owner, $generated_indexes) = @_; + + return Ora2Pg::MySQL::_get_indexes($self,$table,$owner) if ($self->{is_mysql}); + + # Retrieve FTS indexes information before. + my %idx_info = (); + %idx_info = $self->_get_fts_indexes_info($owner) if ($self->_table_exists('CTXSYS', 'CTX_INDEX_VALUES')); + + my $sub_owner = ''; + if ($owner) { + $sub_owner = "AND A.INDEX_OWNER=B.TABLE_OWNER"; + } + + my $condition = ''; + $condition .= "AND A.TABLE_NAME='$table' " if ($table); + if ($owner) { + $condition .= "AND A.INDEX_OWNER='$owner' "; + } else { + $condition .= " AND A.INDEX_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + if (!$table) { + $condition .= $self->limit_to_objects('TABLE|INDEX', "A.TABLE_NAME|A.INDEX_NAME"); + } else { + @{$self->{query_bind_params}} = (); + } + + # When comparing number of index we need to retrieve generated index (mostly PK) + my $generated = ''; + $generated = " B.GENERATED = 'N' AND" if (!$generated_indexes); + + # Retrieve all indexes + my $sth = ''; + if ($self->{db_version} !~ /Release 8/) { + my $no_mview = " AND (A.INDEX_OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + $no_mview .= " AND (A.INDEX_OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + $no_mview = '' if ($self->{type} eq 'MVIEW'); + $sth = $self->{dbh}->prepare(<logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); +SELECT DISTINCT A.INDEX_NAME,A.COLUMN_NAME,B.UNIQUENESS,A.COLUMN_POSITION,B.INDEX_TYPE,B.TABLE_TYPE,B.GENERATED,B.JOIN_INDEX,A.TABLE_NAME,A.INDEX_OWNER,B.TABLESPACE_NAME,B.ITYP_NAME,B.PARAMETERS,A.DESCEND +FROM $self->{prefix}_IND_COLUMNS A +JOIN $self->{prefix}_INDEXES B ON (B.INDEX_NAME=A.INDEX_NAME AND B.OWNER=A.INDEX_OWNER) +WHERE$generated B.TEMPORARY = 'N' $condition $no_mview +ORDER BY A.COLUMN_POSITION +END + } else { + # an 8i database. + $sth = $self->{dbh}->prepare(<logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); +SELECT DISTINCT A.INDEX_NAME,A.COLUMN_NAME,B.UNIQUENESS,A.COLUMN_POSITION,B.INDEX_TYPE,B.TABLE_TYPE,B.GENERATED, 'NO', A.TABLE_NAME,A.INDEX_OWNER,B.TABLESPACE_NAME,B.ITYP_NAME,B.PARAMETERS,A.DESCEND +FROM $self->{prefix}_IND_COLUMNS A, $self->{prefix}_INDEXES B +WHERE B.INDEX_NAME=A.INDEX_NAME AND B.OWNER=A.INDEX_OWNER $condition +AND$generated B.TEMPORARY = 'N' +ORDER BY A.COLUMN_POSITION +END + } + + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my $idxnc = qq{SELECT IE.COLUMN_EXPRESSION FROM $self->{prefix}_IND_EXPRESSIONS IE, $self->{prefix}_IND_COLUMNS IC +WHERE IE.INDEX_OWNER = IC.INDEX_OWNER +AND IE.INDEX_NAME = IC.INDEX_NAME +AND IE.TABLE_OWNER = IC.TABLE_OWNER +AND IE.TABLE_NAME = IC.TABLE_NAME +AND IE.COLUMN_POSITION = IC.COLUMN_POSITION +AND IC.COLUMN_NAME = ? +AND IE.TABLE_NAME = ? +AND IC.TABLE_OWNER = ? +}; + my $sth2 = $self->{dbh}->prepare($idxnc); + my %data = (); + my %unique = (); + my %idx_type = (); + while (my $row = $sth->fetch) + { + # Exclude log indexes of materialized views, there must be a better + # way to exclude then than looking at index name, fill free to fix it. + next if ($row->[0] =~ /^I_SNAP\$_/); + + my $save_tb = $row->[8]; + if (!$self->{schema} && $self->{export_schema}) { + $row->[8] = "$row->[9].$row->[8]"; + } + if (!$self->{preserve_case}) { + next if (exists $self->{modify}{"\L$row->[8]\E"} && !grep(/^\Q$row->[1]\E$/i, @{$self->{modify}{"\L$row->[8]\E"}})); + } else { + next if (exists $self->{modify}{$row->[8]} && !grep(/^\Q$row->[1]\E$/i, @{$self->{modify}{$row->[8]}})); + } + # Show a warning when an index has the same name as the table + if ( !$self->{indexes_renaming} && !$self->{indexes_suffix} && (lc($row->[0]) eq lc($table)) ) { + print STDERR "WARNING: index $row->[0] has the same name as the table itself. Please rename it before export or enable INDEXES_RENAMING.\n"; + } + $unique{$row->[8]}{$row->[0]} = $row->[2]; + + # Save original column name + my $colname = $row->[1]; + # Replace function based index type + if ( ($row->[4] =~ /FUNCTION-BASED/i) && ($colname =~ /^SYS_NC\d+\$$/) ) + { + $sth2->execute($colname,$save_tb,$row->[-5]) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my $nc = $sth2->fetch(); + $row->[1] = $nc->[0]; + $row->[1] =~ s/"//g; + $row->[1] =~ s/'//g if ($row->[1] =~ /^'[^'\s]+'$/); + # Single row constraint based on a constant and a function based unique index + if ($row->[2] eq 'UNIQUE' && $nc->[0] =~ /^\d+$/ && $row->[4] =~ /FUNCTION-BASED/i) { + $row->[1] = '(' . $nc->[0] . ')'; + } + # Enclose with double quote if required when is is not an index function + elsif ($row->[1] !~ /\(.*\)/ && $row->[4] !~ /FUNCTION-BASED/i) { + $row->[1] = $self->quote_object_name($row->[1]); + } + # Append DESC sort order when not default to ASC + if ($row->[13] eq 'DESC') { + $row->[1] .= " DESC"; + } + } + else + { + # Quote column with unsupported symbols + $row->[1] = $self->quote_object_name($row->[1]); + } + + $row->[1] =~ s/SYS_EXTRACT_UTC\s*\(([^\)]+)\)/$1/isg; + + # Index with DESC are declared as FUNCTION-BASED, fix that + if (($row->[4] =~ /FUNCTION-BASED/i) && ($row->[1] !~ /\(.*\)/)) { + $row->[4] =~ s/FUNCTION-BASED\s*//; + } + $idx_type{$row->[8]}{$row->[0]}{type_name} = $row->[11]; + if (($#{$row} > 6) && ($row->[7] eq 'Y')) { + $idx_type{$row->[8]}{$row->[0]}{type} = $row->[4] . ' JOIN'; + } else { + $idx_type{$row->[8]}{$row->[0]}{type} = $row->[4]; + } + my $idx_name = $row->[0]; + if (!$self->{schema} && $self->{export_schema}) { + $idx_name = "$row->[9].$row->[0]"; + } + if (exists $idx_info{$idx_name}) { + $idx_type{$row->[8]}{$row->[0]}{stemmer} = $idx_info{$idx_name}{stemmer}; + } + if ($row->[11] =~ /SPATIAL_INDEX/) { + $idx_type{$row->[8]}{$row->[0]}{type} = 'SPATIAL INDEX'; + if ($row->[12] =~ /layer_gtype=([^\s,]+)/i) { + $idx_type{$row->[9]}{$row->[0]}{type_constraint} = uc($1); + } + if ($row->[12] =~ /sdo_indx_dims=(\d+)/i) { + $idx_type{$row->[8]}{$row->[0]}{type_dims} = $1; + } + } + if ($row->[4] eq 'BITMAP') { + $idx_type{$row->[8]}{$row->[0]}{type} = $row->[4]; + } + push(@{$data{$row->[8]}{$row->[0]}}, $row->[1]); + $index_tablespace{$row->[8]}{$row->[0]} = $row->[10]; + + } + $sth->finish(); + $sth2->finish(); + + return \%unique, \%data, \%idx_type, \%index_tablespace; +} + +=head2 _get_fts_indexes_info + +This function retrieve FTS index attributes informations + +Returns a hash of containing all useful attribute values for all FTS indexes + +=cut + +sub _get_fts_indexes_info +{ + my ($self, $owner) = @_; + + my $condition = ''; + $condition .= "AND IXV_INDEX_OWNER='$owner' " if ($owner); + $condition .= $self->limit_to_objects('INDEX', "IXV_INDEX_NAME"); + + # Retrieve all indexes informations + my $sth = $self->{dbh}->prepare(<logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); +SELECT DISTINCT IXV_INDEX_OWNER,IXV_INDEX_NAME,IXV_CLASS,IXV_ATTRIBUTE,IXV_VALUE +FROM CTXSYS.CTX_INDEX_VALUES +WHERE (IXV_CLASS='WORDLIST' AND IXV_ATTRIBUTE='STEMMER') $condition +END + + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my %indexes_info = (); + while (my $row = $sth->fetch) { + my $save_idx = $row->[1]; + if (!$self->{schema} && $self->{export_schema}) { + $row->[1] = "$row->[0].$row->[1]"; + } + $indexes_info{$row->[1]}{"\L$row->[3]\E"} = $row->[4]; + } + + return %indexes_info; +} + + + +=head2 _get_sequences + +This function implements an Oracle-native sequences information. + +Returns a hash of an array of sequence names with MIN_VALUE, MAX_VALUE, +INCREMENT and LAST_NUMBER for the specified table. + +=cut + +sub _get_sequences +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_sequences($self) if ($self->{is_mysql}); + + # Retrieve all indexes + my $str = "SELECT DISTINCT SEQUENCE_NAME, MIN_VALUE, MAX_VALUE, INCREMENT_BY, LAST_NUMBER, CACHE_SIZE, CYCLE_FLAG, SEQUENCE_OWNER FROM $self->{prefix}_SEQUENCES"; + if (!$self->{schema}) { + $str .= " WHERE SEQUENCE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE SEQUENCE_OWNER = '$self->{schema}'"; + } + # Exclude sequence used for IDENTITY columns + $str .= " AND SEQUENCE_NAME NOT LIKE 'ISEQ\$\$_%'"; + $str .= $self->limit_to_objects('SEQUENCE', 'SEQUENCE_NAME'); + #$str .= " ORDER BY SEQUENCE_NAME"; + + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my @seqs = (); + while (my $row = $sth->fetch) + { + push(@seqs, [ @$row ]); + } + + return \@seqs; +} + +=head2 _get_identities + +This function retrieve information about IDENTITY columns that must be +exported as PostgreSQL serial. + +=cut + +sub _get_identities +{ + my ($self) = @_; + + return Ora2Pg::MySQL::_get_identities($self) if ($self->{is_mysql}); + + # Identity column appears in version 12 only + return if ($self->{db_version} =~ /Release (8|9|10|11)/); + + # Retrieve all indexes + my $str = "SELECT OWNER, TABLE_NAME, COLUMN_NAME, GENERATION_TYPE, IDENTITY_OPTIONS FROM $self->{prefix}_TAB_IDENTITY_COLS"; + if (!$self->{schema}) { + $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE OWNER = '$self->{schema}'"; + } + $str .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); + #$str .= " ORDER BY OWNER, TABLE_NAME, COLUMN_NAME"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %seqs = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[1] = "$row->[0].$row->[1]"; + } + # GENERATION_TYPE can be ALWAYS, BY DEFAULT and BY DEFAULT ON NULL + $seqs{$row->[1]}{$row->[2]}{generation} = $row->[3]; + # SEQUENCE options + $seqs{$row->[1]}{$row->[2]}{options} = $row->[4]; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/(SCALE|EXTEND|SESSION)_FLAG: .//ig; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/KEEP_VALUE: .//is; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/(START WITH):/$1/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/(INCREMENT BY):/$1/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/MAX_VALUE:/MAXVALUE/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/MIN_VALUE:/MINVALUE/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/CYCLE_FLAG: N/NO CYCLE/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/CYCLE_FLAG: Y/CYCLE/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/CACHE_SIZE:/CACHE/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/CACHE_SIZE:/CACHE/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/ORDER_FLAG: .//; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/,//g; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/\s$//; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/CACHE\s+0/CACHE 1/; + # For default values don't use option at all + if ( $seqs{$row->[1]}{$row->[2]}{options} eq 'START WITH 1 INCREMENT BY 1 MAXVALUE 9999999999999999999999999999 MINVALUE 1 NO CYCLE CACHE 20') { + delete $seqs{$row->[1]}{$row->[2]}{options}; + } + # Limit the sequence value to bigint max + $seqs{$row->[1]}{$row->[2]}{options} =~ s/MAXVALUE 9999999999999999999999999999/MAXVALUE 9223372036854775807/; + $seqs{$row->[1]}{$row->[2]}{options} =~ s/\s+/ /g; + } + + return %seqs; +} + +=head2 _get_external_tables + +This function implements an Oracle-native external tables information. + +Returns a hash of external tables names with the file they are based on. + +=cut + +sub _get_external_tables +{ + my($self) = @_; + + # Retrieve all database link from dba_db_links table + my $str = "SELECT a.*,b.DIRECTORY_PATH,c.LOCATION,a.OWNER FROM $self->{prefix}_EXTERNAL_TABLES a, $self->{prefix}_DIRECTORIES b, $self->{prefix}_EXTERNAL_LOCATIONS c"; + if (!$self->{schema}) { + $str .= " WHERE a.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE a.OWNER = '$self->{schema}'"; + } + $str .= " AND a.DEFAULT_DIRECTORY_NAME = b.DIRECTORY_NAME AND a.TABLE_NAME=c.TABLE_NAME AND a.DEFAULT_DIRECTORY_NAME=c.DIRECTORY_NAME AND a.OWNER=c.OWNER"; + $str .= $self->limit_to_objects('TABLE', 'a.TABLE_NAME'); + #$str .= " ORDER BY a.TABLE_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[1] = "$row->[0].$row->[1]"; + } + $data{$row->[1]}{directory} = $row->[5]; + $data{$row->[1]}{directory_path} = $row->[10]; + if ($data{$row->[1]}{directory_path} =~ /([\/\\])/) { + $data{$row->[1]}{directory_path} .= $1 if ($data{$row->[1]}{directory_path} !~ /$1$/); + } + $data{$row->[1]}{location} = $row->[11]; + $data{$row->[1]}{delimiter} = ','; + if ($row->[8] =~ /FIELDS TERMINATED BY '(.)'/is) { + $data{$row->[1]}{delimiter} = $1; + } + if ($row->[8] =~ /PREPROCESSOR EXECDIR\s*:\s*'([^']+)'/is) { + $data{$row->[1]}{program} = $1; + } + } + $sth->finish(); + + return %data; +} + +=head2 _get_directory + +This function implements an Oracle-native directory information. + +Returns a hash of directory names with the path they are based on. + +=cut + +sub _get_directory +{ + my ($self) = @_; + + # Retrieve all database link from dba_db_links table + my $str = "SELECT d.DIRECTORY_NAME, d.DIRECTORY_PATH, d.OWNER, p.GRANTEE, p.PRIVILEGE FROM $self->{prefix}_DIRECTORIES d, $self->{prefix}_TAB_PRIVS p"; + $str .= " WHERE d.DIRECTORY_NAME = p.TABLE_NAME"; + if (!$self->{schema}) { + $str .= " AND p.GRANTEE NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND p.GRANTEE = '$self->{schema}'"; + } + $str .= $self->limit_to_objects('TABLE', 'd.DIRECTORY_NAME'); + #$str .= " ORDER BY d.DIRECTORY_NAME"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[2].$row->[0]"; + } + $data{$row->[0]}{path} = $row->[1]; + if ($row->[1] !~ /\/$/) { + $data{$row->[0]}{path} .= '/'; + } + $data{$row->[0]}{grantee}{$row->[3]} .= $row->[4]; + } + $sth->finish(); + + return %data; +} + +=head2 _get_dblink + +This function implements an Oracle-native database link information. + +Returns a hash of dblink names with the connection they are based on. + +=cut + + +sub _get_dblink +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_dblink($self) if ($self->{is_mysql}); + + # Retrieve all database link from dba_db_links table + my $str = "SELECT OWNER,DB_LINK,USERNAME,HOST FROM $self->{prefix}_DB_LINKS"; + if (!$self->{schema}) { + $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE OWNER = '$self->{schema}'"; + } + $str .= $self->limit_to_objects('DBLINK', 'DB_LINK'); + #$str .= " ORDER BY DB_LINK"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[1] = "$row->[0].$row->[1]"; + } + $data{$row->[1]}{owner} = $row->[0]; + $data{$row->[1]}{user} = $row->[2]; + $data{$row->[1]}{username} = $self->{pg_user} || $row->[2]; + $data{$row->[1]}{host} = $row->[3]; + } + + return %data; +} + +=head2 _get_job + +This function implements an Oracle-native job information. + +Reads together from view [ALL|DBA]_JOBS and from view [ALL|DBA]_SCHEDULER_JOBS. + +Returns a hash of job number with the connection they are based on. + +=cut + + +sub _get_job +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_job($self) if ($self->{is_mysql}); + + # Jobs appears in version 10 only + return if ($self->{db_version} =~ /Release [8|9]/); + + # Retrieve all database job from user_jobs table + my $str = "SELECT JOB,WHAT,INTERVAL,SCHEMA_USER FROM $self->{prefix}_JOBS"; + if (!$self->{schema}) { + $str .= " WHERE SCHEMA_USER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE SCHEMA_USER = '$self->{schema}'"; + } + $str .= $self->limit_to_objects('JOB', 'JOB'); + #$str .= " ORDER BY JOB"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[3].$row->[0]"; + } + $data{$row->[0]}{what} = $row->[1]; + $data{$row->[0]}{interval} = $row->[2]; + } + + # Retrieve all database jobs from view [ALL|DBA]_SCHEDULER_JOBS + $str = "SELECT job_name AS JOB, job_action AS WHAT, repeat_interval AS INTERVAL, owner AS SCHEMA_USER"; + $str .= " FROM $self->{prefix}_SCHEDULER_JOBS"; + $str .= " WHERE repeat_interval IS NOT NULL"; + $str .= " AND client_id IS NULL"; + if (!$self->{schema}) { + $str .= " AND owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND owner = '$self->{schema}'"; + } + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ($row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[3].$row->[0]"; + } + $data{$row->[0]}{what} = $row->[1]; + $data{$row->[0]}{interval} = $row->[2]; + } + + return %data; +} + + +=head2 _get_views + +This function implements an Oracle-native views information. + +Returns a hash of view names with the SQL queries they are based on. + +=cut + +sub _get_views +{ + my($self) = @_; + + + return Ora2Pg::MySQL::_get_views($self) if ($self->{is_mysql}); + + my $owner = ''; + if ($self->{schema}) { + $owner = "AND A.OWNER='$self->{schema}' "; + } else { + $owner = "AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + + my %comments = (); + if ($self->{type} ne 'SHOW_REPORT') + { + my $sql = "SELECT A.TABLE_NAME,A.COMMENTS,A.TABLE_TYPE,A.OWNER FROM ALL_TAB_COMMENTS A, ALL_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='VIEW' $owner"; + $sql .= $self->limit_to_objects('VIEW', 'A.TABLE_NAME'); + my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) + { + if (!$self->{schema} && $self->{export_schema}) + { + $row->[0] = "$row->[3].$row->[0]"; + } + $comments{$row->[0]}{comment} = $row->[1]; + $comments{$row->[0]}{table_type} = $row->[2]; + } + $sth->finish(); + } + + # Retrieve all views + my $str = "SELECT v.VIEW_NAME,v.TEXT,v.OWNER FROM $self->{prefix}_VIEWS v"; + if (!$self->{export_invalid}) { + $str .= ", $self->{prefix}_OBJECTS a"; + } + + if (!$self->{schema}) { + $str .= " WHERE v.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE v.OWNER = '$self->{schema}'"; + } + + if (!$self->{export_invalid}) { + $str .= " AND a.OBJECT_TYPE='VIEW' AND a.STATUS='VALID' AND v.VIEW_NAME=a.OBJECT_NAME AND a.OWNER=v.OWNER"; + } + $str .= $self->limit_to_objects('VIEW', 'v.VIEW_NAME'); + #$str .= " ORDER BY v.OWNER,v.VIEW_NAME"; + + + # Compute view order, where depended view appear before using view + my %view_order = (); + if ($self->{type} ne 'SHOW_REPORT' && !$self->{no_view_ordering}) + { + if ($self->{db_version} !~ /Release (8|9|10|11\.1)/) + { + if ($self->{schema}) { + $owner = "AND o.OWNER='$self->{schema}' "; + } else { + $owner = "AND o.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + $sql = qq{ +WITH x (ITER, OWNER, OBJECT_NAME) AS +( SELECT 1 , o.OWNER, o.OBJECT_NAME FROM ALL_OBJECTS o WHERE OBJECT_TYPE = 'VIEW' $owner + AND NOT EXISTS (SELECT 1 FROM ALL_DEPENDENCIES d WHERE TYPE LIKE 'VIEW' AND REFERENCED_TYPE = 'VIEW' + AND REFERENCED_OWNER = o.OWNER AND d.OWNER = o.OWNER and o.OBJECT_NAME=d.NAME) +UNION ALL + SELECT ITER + 1, d.OWNER, d.NAME FROM ALL_DEPENDENCIES d + JOIN x ON d.REFERENCED_OWNER = x.OWNER and d.REFERENCED_NAME = x.OBJECT_NAME + WHERE TYPE LIKE 'VIEW' AND REFERENCED_TYPE = 'VIEW' +) +SELECT max(ITER) ITER, OWNER, OBJECT_NAME FROM x +GROUP BY OWNER, OBJECT_NAME +ORDER BY ITER ASC, 2, 3 +}; + + my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + $view_order{"\U$row->[1].$row->[2]\E"} = $row->[0]; + } + $sth->finish(); + } + } + + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) + { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[2].$row->[0]"; + } + $data{$row->[0]}{text} = $row->[1]; + $data{$row->[0]}{owner} = $row->[2]; + $data{$row->[0]}{comment} = $comments{$row->[0]}{comment} || ''; + if ($self->{type} ne 'SHOW_REPORT') + { + @{$data{$row->[0]}{alias}} = $self->_alias_info ($row->[0]); + } + if ($self->{type} ne 'SHOW_REPORT' && exists $view_order{"\U$row->[2].$row->[0]\E"}) + { + $data{$row->[0]}{iter} = $view_order{"\U$row->[2].$row->[0]\E"}; + } + } + + return %data; +} + +=head2 _get_materialized_views + +This function implements an Oracle-native materialized views information. + +Returns a hash of view names with the SQL queries they are based on. + +=cut + +sub _get_materialized_views +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_materialized_views($self) if ($self->{is_mysql}); + + # Retrieve all views + my $str = "SELECT MVIEW_NAME,QUERY,UPDATABLE,REFRESH_MODE,REFRESH_METHOD,USE_NO_INDEX,REWRITE_ENABLED,BUILD_MODE,OWNER FROM $self->{prefix}_MVIEWS"; + if ($self->{db_version} =~ /Release 8/) { + $str = "SELECT MVIEW_NAME,QUERY,UPDATABLE,REFRESH_MODE,REFRESH_METHOD,'',REWRITE_ENABLED,BUILD_MODE,OWNER FROM $self->{prefix}_MVIEWS"; + } + if (!$self->{schema}) { + $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE OWNER = '$self->{schema}'"; + } + $str .= $self->limit_to_objects('MVIEW', 'MVIEW_NAME'); + #$str .= " ORDER BY MVIEW_NAME"; + my $sth = $self->{dbh}->prepare($str); + if (not defined $sth) { + $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + if (not $sth->execute(@{$self->{query_bind_params}})) { + $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + return (); + } + + my %data = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[8].$row->[0]"; + } + $data{$row->[0]}{text} = $row->[1]; + $data{$row->[0]}{updatable} = ($row->[2] eq 'Y') ? 1 : 0; + $data{$row->[0]}{refresh_mode} = $row->[3]; + $data{$row->[0]}{refresh_method} = $row->[4]; + $data{$row->[0]}{no_index} = ($row->[5] eq 'Y') ? 1 : 0; + $data{$row->[0]}{rewritable} = ($row->[6] eq 'Y') ? 1 : 0; + $data{$row->[0]}{build_mode} = $row->[7]; + $data{$row->[0]}{owner} = $row->[8]; + } + + return %data; +} + +sub _get_materialized_view_names +{ + my($self) = @_; + + # Retrieve all views + my $str = "SELECT MVIEW_NAME,OWNER FROM $self->{prefix}_MVIEWS"; + if (!$self->{schema}) { + $str .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " WHERE OWNER = '$self->{schema}'"; + } + $str .= $self->limit_to_objects('MVIEW', 'MVIEW_NAME'); + #$str .= " ORDER BY MVIEW_NAME"; + my $sth = $self->{dbh}->prepare($str); + if (not defined $sth) { + $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + if (not $sth->execute(@{$self->{query_bind_params}})) { + $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + + my @data = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[1].$row->[0]"; + } + push(@data, uc($row->[0])); + } + + return @data; +} + + +=head2 _alias_info + +This function implements an Oracle-native column information. + +Returns a list of array references containing the following information +for each alias of the specified view: + +[( + column name, + column id +)] + +=cut + +sub _alias_info +{ + my ($self, $view) = @_; + + my $str = "SELECT COLUMN_NAME, COLUMN_ID, OWNER FROM $self->{prefix}_TAB_COLUMNS WHERE TABLE_NAME='$view'"; + if ($self->{schema}) { + $str .= " AND OWNER = '$self->{schema}'"; + } else { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + $str .= " ORDER BY COLUMN_ID ASC"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my $data = $sth->fetchall_arrayref(); + $self->logit("View $view column aliases:\n", 1); + foreach my $d (@$data) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[2].$row->[0]"; + } + $self->logit("\t$d->[0] => column id:$d->[1]\n", 1); + } + + return @$data; + +} + +=head2 _get_triggers + +This function implements an Oracle-native triggers information. + +Returns an array of refarray of all triggers information. + +=cut + +sub _get_triggers +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_triggers($self) if ($self->{is_mysql}); + + # Retrieve all indexes + my $str = "SELECT TRIGGER_NAME, TRIGGER_TYPE, TRIGGERING_EVENT, TABLE_NAME, TRIGGER_BODY, WHEN_CLAUSE, DESCRIPTION, ACTION_TYPE, OWNER FROM $self->{prefix}_TRIGGERS WHERE STATUS='ENABLED'"; + if (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND OWNER = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','TABLE_NAME|TABLE_NAME|TRIGGER_NAME'); + + #$str .= " ORDER BY TABLE_NAME, TRIGGER_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my @triggers = (); + while (my $row = $sth->fetch) { + push(@triggers, [ @$row ]); + } + + return \@triggers; +} + +sub _list_triggers +{ + my($self) = @_; + + return Ora2Pg::MySQL::_list_triggers($self) if ($self->{is_mysql}); + + # Retrieve all indexes + my $str = "SELECT TRIGGER_NAME, TABLE_NAME, OWNER FROM $self->{prefix}_TRIGGERS WHERE STATUS='ENABLED'"; + if (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND OWNER = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','TABLE_NAME|TABLE_NAME|TRIGGER_NAME'); + + #$str .= " ORDER BY TABLE_NAME, TRIGGER_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %triggers = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + push(@{$triggers{"$row->[2].$row->[1]"}}, $row->[0]); + } else { + push(@{$triggers{$row->[1]}}, $row->[0]); + } + } + + return %triggers; +} + +=head2 _get_plsql_metadata + +This function retrieve all metadata on Oracle store procedure. + +Returns a hash of all function names with their metadata +information (arguments, return type, etc.). + +=cut + +sub _get_plsql_metadata +{ + my $self = shift; + my $owner = shift; + + return Ora2Pg::MySQL::_get_plsql_metadata($self, $owner) if ($self->{is_mysql}); + + # Retrieve all functions + my $str = "SELECT DISTINCT OBJECT_NAME,OWNER,OBJECT_TYPE FROM $self->{prefix}_OBJECTS WHERE (OBJECT_TYPE = 'FUNCTION' OR OBJECT_TYPE = 'PROCEDURE' OR OBJECT_TYPE = 'PACKAGE BODY')"; + $str .= " AND STATUS='VALID'" if (!$self->{export_invalid}); + if ($owner) { + $str .= " AND OWNER = '$owner'"; + $self->logit("Looking forward functions declaration in schema $owner.\n", 1) if (!$quiet); + } elsif (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + $self->logit("Looking forward functions declaration in all schema.\n", 1) if (!$quiet); + } else { + $str .= " AND OWNER = '$self->{schema}'"; + $self->logit("Looking forward functions declaration in schema $self->{schema}.\n", 1) if (!$quiet); + } + #$str .= " ORDER BY OBJECT_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %functions = (); + my @fct_done = (); + push(@fct_done, @EXCLUDED_FUNCTION); + while (my $row = $sth->fetch) { + next if (grep(/^$row->[1].$row->[0]$/i, @fct_done)); + push(@fct_done, "$row->[1].$row->[0]"); + $self->{function_metadata}{$row->[1]}{'none'}{$row->[0]}{type} = $row->[2]; + } + $sth->finish(); + + # Get content of package body + my $sql = "SELECT NAME, OWNER, TYPE, TEXT FROM $self->{prefix}_SOURCE"; + if ($owner) { + $sql .= " WHERE OWNER = '$owner'"; + } elsif (!$self->{schema}) { + $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $sql .= " WHERE OWNER = '$self->{schema}'"; + } + $sql .= " AND TYPE <> 'PACKAGE'"; + $sql .= " ORDER BY OWNER, NAME, LINE"; + $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) + { + next if (!exists $self->{function_metadata}{$row->[1]}{'none'}{$row->[0]}); + $self->{function_metadata}{$row->[1]}{'none'}{$row->[0]}{text} .= $row->[3]; + } + $sth->finish(); + + # For each schema in the Oracle instance + foreach my $sch (sort keys %{ $self->{function_metadata} }) + { + next if ( ($owner && ($sch ne $owner)) || (!$owner && $self->{schema} && ($sch ne $self->{schema})) ); + # Look for functions/procedures + foreach my $name (sort keys %{$self->{function_metadata}{$sch}{'none'}}) + { + if ($self->{function_metadata}{$sch}{'none'}{$name}{type} ne 'PACKAGE BODY') + { + # Retrieve metadata for this function after removing comments + $self->_remove_comments(\$self->{function_metadata}{$sch}{'none'}{$name}{text}, 1); + $self->{comment_values} = (); + $self->{function_metadata}{$sch}{'none'}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; + my %fct_detail = $self->_lookup_function($self->{function_metadata}{$sch}{'none'}{$name}{text}); + if (!exists $fct_detail{name}) + { + delete $self->{function_metadata}{$sch}{'none'}{$name}; + next; + } + delete $fct_detail{code}; + delete $fct_detail{before}; + %{$self->{function_metadata}{$sch}{'none'}{$name}{metadata}} = %fct_detail; + delete $self->{function_metadata}{$sch}{'none'}{$name}{text}; + } + else + { + $self->_remove_comments(\$self->{function_metadata}{$sch}{'none'}{$name}{text}, 1); + $self->{comment_values} = (); + $self->{function_metadata}{$sch}{'none'}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; + my %infos = $self->_lookup_package($self->{function_metadata}{$sch}{'none'}{$name}{text}); + delete $self->{function_metadata}{$sch}{'none'}{$name}; + $name =~ s/"//g; + foreach my $f (sort keys %infos) + { + next if (!$f); + my $fn = lc($f); + delete $infos{$f}{code}; + delete $infos{$f}{before}; + %{$self->{function_metadata}{$sch}{$name}{$fn}{metadata}} = %{$infos{$f}}; + my $res_name = $f; + $res_name =~ s/^([^\.]+)\.//; + $f =~ s/^([^\.]+)\.//; + if ($self->{package_as_schema}) { + $res_name = $name . '.' . $res_name; + } else { + $res_name = $name . '_' . $res_name; + } + $res_name =~ s/"_"/_/g; + $f =~ s/"//g; + $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{name} = $self->quote_object_name($res_name); + $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{package} = $name; + } + } + } + } +} + + +=head2 _get_package_function_list + +This function retrieve all function and procedure +defined on Oracle store procedure PACKAGE. + +Returns a hash of all package function names + +=cut + +sub _get_package_function_list +{ + my $self = shift; + my $owner = shift; + + return Ora2Pg::MySQL::_get_package_function_list($self, $owner) if ($self->{is_mysql}); + + # Retrieve all package information + my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE = 'PACKAGE BODY'"; + $str .= " AND STATUS='VALID'" if (!$self->{export_invalid}); + if ($owner) { + $str .= " AND OWNER = '$owner'"; + $self->logit("Looking forward functions declaration in schema $owner.\n", 1) if (!$quiet); + } elsif (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + $self->logit("Looking forward functions declaration in all schema.\n", 1) if (!$quiet); + } else { + $str .= " AND OWNER = '$self->{schema}'"; + $self->logit("Looking forward functions declaration in schema $self->{schema}.\n", 1) if (!$quiet); + } + #$str .= " ORDER BY OBJECT_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my @packages = (); + while (my $row = $sth->fetch) { + next if (grep(/^$row->[0]$/i, @packages)); + push(@packages, $row->[0]); + } + $sth->finish(); + + # Get content of all packages definition + my $sql = "SELECT NAME, OWNER, TYPE, TEXT FROM $self->{prefix}_SOURCE"; + if ($owner) { + $sql .= " WHERE OWNER = '$owner'"; + } elsif (!$self->{schema}) { + $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $sql .= " WHERE OWNER = '$self->{schema}'"; + } + $sql .= " AND TYPE <> 'PACKAGE'"; + $sql .= " AND NAME IN ('" . join("','", @packages) . "')" if ($#packages >= 0); + $sql .= " ORDER BY OWNER, NAME, LINE"; + $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); + my %function_metadata = (); + while (my $row = $sth->fetch) { + $function_metadata{$row->[1]}{$row->[0]}{text} .= $row->[3]; + } + $sth->finish(); + + my @fct_done = (); + push(@fct_done, @EXCLUDED_FUNCTION); + foreach my $sch (sort keys %function_metadata) { + next if ( ($owner && ($sch ne $owner)) || (!$owner && $self->{schema} && ($sch ne $self->{schema})) ); + foreach my $name (sort keys %{$function_metadata{$sch}}) { + $self->_remove_comments(\$function_metadata{$sch}{$name}{text}, 1); + $self->{comment_values} = (); + $function_metadata{$sch}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; + my %infos = $self->_lookup_package($function_metadata{$sch}{$name}{text}); + delete $function_metadata{$sch}{$name}; + foreach my $f (sort keys %infos) { + next if (!$f); + my $fn = lc($f); + my $res_name = $f; + if ($res_name =~ s/^([^\.]+)\.//) { + next if (lc($1) ne lc($name)); + } + if ($self->{package_as_schema}) { + $res_name = $name . '.' . $res_name; + } else { + $res_name = $name . '_' . $res_name; + } + $res_name =~ s/"_"/_/g; + $f =~ s/"//gs; + if ($res_name) { + $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{name} = $self->quote_object_name($res_name); + $self->{package_functions}{"\L$name\E"}{"\L$f\E"}{package} = $name; + } + } + } + } +} + +=head2 _get_functions + +This function implements an Oracle-native functions information. + +Returns a hash of all function names with their PLSQL code. + +=cut + +sub _get_functions +{ + my $self = shift; + + return Ora2Pg::MySQL::_get_functions($self) if ($self->{is_mysql}); + + # Retrieve all functions + my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='FUNCTION'"; + $str .= " AND STATUS='VALID'" if (!$self->{export_invalid}); + if (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND OWNER = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('FUNCTION','OBJECT_NAME'); + #$str .= " ORDER BY OBJECT_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %functions = (); + my @fct_done = (); + push(@fct_done, @EXCLUDED_FUNCTION); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[1].$row->[0]"; + } + next if (grep(/^$row->[0]$/i, @fct_done)); + push(@fct_done, $row->[0]); + $functions{"$row->[0]"}{owner} = $row->[1]; + } + $sth->finish(); + + my $sql = "SELECT NAME,OWNER,TEXT FROM $self->{prefix}_SOURCE"; + if (!$self->{schema}) { + $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $sql .= " WHERE OWNER = '$self->{schema}'"; + } + $sql .= " " . $self->limit_to_objects('FUNCTION','NAME'); + $sql .= " ORDER BY OWNER,NAME,LINE"; + $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[1].$row->[0]"; + } + # Remove some bargage when migrating from 8i + $row->[2] =~ s/\bAUTHID\s+[^\s]+\s+//is; + if (exists $functions{"$row->[0]"}) { + $functions{"$row->[0]"}{text} .= $row->[2]; + } + } + + return \%functions; +} + +sub _get_functions2 +{ + my $self = shift; + + return Ora2Pg::MySQL::_get_functions($self) if ($self->{is_mysql}); + + # Retrieve all functions + my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='FUNCTION'"; + $str .= " AND STATUS='VALID'" if (!$self->{export_invalid}); + if (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND OWNER = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('FUNCTION','OBJECT_NAME'); + #$str .= " ORDER BY OBJECT_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %functions = (); + my @fct_done = (); + push(@fct_done, @EXCLUDED_FUNCTION); + while (my $row = $sth->fetch) { + my $sql = "SELECT TEXT FROM $self->{prefix}_SOURCE WHERE OWNER='$row->[1]' AND NAME='$row->[0]' ORDER BY LINE"; + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[1].$row->[0]"; + } + next if (grep(/^$row->[0]$/i, @fct_done)); + push(@fct_done, $row->[0]); + my $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth2->execute or $self->logit("FATAL: " . $sth2->errstr . "\n", 0, 1); + while (my $r = $sth2->fetch) { + $functions{"$row->[0]"}{text} .= $r->[0]; + } + $functions{"$row->[0]"}{owner} .= $row->[1]; + } + + return \%functions; +} + +=head2 _get_procedures + +This procedure implements an Oracle-native procedures information. + +Returns a hash of all procedure names with their PLSQL code. + +=cut + +sub _get_procedures +{ + my $self = shift; + + return Ora2Pg::MySQL::_get_functions($self) if ($self->{is_mysql}); + + # Retrieve all functions + my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='PROCEDURE'"; + $str .= " AND STATUS='VALID'" if (!$self->{export_invalid}); + if (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND OWNER = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('PROCEDURE','OBJECT_NAME'); + #$str .= " ORDER BY OBJECT_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %procedures = (); + my @fct_done = (); + push(@fct_done, @EXCLUDED_FUNCTION); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[1].$row->[0]"; + } + next if (grep(/^$row->[0]$/i, @fct_done)); + push(@fct_done, $row->[0]); + $procedures{"$row->[0]"}{owner} = $row->[1]; + } + $sth->finish(); + + my $sql = "SELECT NAME,OWNER,TEXT FROM $self->{prefix}_SOURCE"; + if (!$self->{schema}) { + $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $sql .= " WHERE OWNER = '$self->{schema}'"; + } + $sql .= " " . $self->limit_to_objects('PROCEDURE','NAME'); + $sql .= " ORDER BY OWNER,NAME,LINE"; + $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[1].$row->[0]"; + } + # Remove some bargage when migrating from 8i + $row->[2] =~ s/\bAUTHID\s+[^\s]+\s+//is; + if (exists $procedures{"$row->[0]"}) { + $procedures{"$row->[0]"}{text} .= $row->[2]; + } + } + + return \%procedures; +} + +=head2 _get_packages + +This function implements an Oracle-native packages information. + +Returns a hash of all package names with their PLSQL code. + +=cut + +sub _get_packages +{ + my ($self) = @_; + + # Retrieve all indexes + #my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE = 'PACKAGE BODY'"; + my $str = "SELECT DISTINCT OBJECT_NAME,OWNER FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE = 'PACKAGE'"; + $str .= " AND STATUS='VALID'" if (!$self->{export_invalid}); + if (!$self->{schema}) { + $str .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } else { + $str .= " AND OWNER = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('PACKAGE','OBJECT_NAME'); + #$str .= " ORDER BY OBJECT_NAME"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %packages = (); + my @fct_done = (); + while (my $row = $sth->fetch) + { + $self->logit("\tFound Package: $row->[0]\n", 1); + next if (grep(/^$row->[0]$/, @fct_done)); + push(@fct_done, $row->[0]); + # Get package definition first + my $sql = "SELECT TEXT FROM $self->{prefix}_SOURCE WHERE OWNER='$row->[1]' AND NAME='$row->[0]' AND TYPE='PACKAGE' ORDER BY LINE"; + my $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth2->execute or $self->logit("FATAL: " . $sth2->errstr . "\n", 0, 1); + while (my $r = $sth2->fetch) { + $packages{$row->[0]}{desc} .= 'CREATE OR REPLACE ' if ($r->[0] =~ /^PACKAGE\s+/is); + $packages{$row->[0]}{desc} .= $r->[0]; + } + $sth2->finish(); + $packages{$row->[0]}{desc} .= "\n" if (exists $packages{$row->[0]}); + + # Then package body code + $sql = "SELECT TEXT FROM $self->{prefix}_SOURCE WHERE OWNER='$row->[1]' AND NAME='$row->[0]' AND TYPE='PACKAGE BODY' ORDER BY LINE"; + $sth2 = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth2->execute or $self->logit("FATAL: " . $sth2->errstr . "\n", 0, 1); + while (my $r = $sth2->fetch) { + $packages{$row->[0]}{text} .= 'CREATE OR REPLACE ' if ($r->[0] =~ /^PACKAGE\s+/is); + $packages{$row->[0]}{text} .= $r->[0]; + } + $packages{$row->[0]}{owner} = $row->[1]; + } + + return \%packages; +} + +=head2 _get_types + +This function implements an Oracle custom types information. + +Returns a hash of all type names with their code. + +=cut + +sub _get_types +{ + my ($self, $name) = @_; + + # Retrieve all user defined types + my $str = "SELECT DISTINCT OBJECT_NAME,OWNER,OBJECT_ID FROM $self->{prefix}_OBJECTS WHERE OBJECT_TYPE='TYPE'"; + $str .= " AND STATUS='VALID'" if (!$self->{export_invalid}); + $str .= " AND OBJECT_NAME='$name'" if ($name); + $str .= " AND GENERATED='N'"; + if ($self->{schema}) { + $str .= "AND OWNER='$self->{schema}' "; + } else { + $str .= "AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + if (!$name) { + $str .= $self->limit_to_objects('TYPE', 'OBJECT_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + #$str .= " ORDER BY OBJECT_NAME"; + + # use a separeate connection + my $local_dbh = $self->_oracle_connection(); + + my $sth = $local_dbh->prepare($str) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); + + my @types = (); + my @fct_done = (); + while (my $row = $sth->fetch) + { + next if ($row->[0] =~ /^(SDO_GEOMETRY|ST_|STGEOM_)/); + my $sql = "SELECT TEXT FROM $self->{prefix}_SOURCE WHERE OWNER='$row->[1]' AND NAME='$row->[0]' AND (TYPE='TYPE' OR TYPE='TYPE BODY') ORDER BY TYPE, LINE"; + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[1].$row->[0]"; + } + $self->logit("\tFound Type: $row->[0]\n", 1); + next if (grep(/^$row->[0]$/, @fct_done)); + push(@fct_done, $row->[0]); + my %tmp = (); + my $sth2 = $local_dbh->prepare($sql) or $self->logit("FATAL: " . $local_dbh->errstr . "\n", 0, 1); + $sth2->execute or $self->logit("FATAL: " . $sth2->errstr . "\n", 0, 1); + while (my $r = $sth2->fetch) { + $tmp{code} .= $r->[0]; + } + $sth2->finish(); + $tmp{name} = $row->[0]; + $tmp{owner} = $row->[1]; + $tmp{pos} = $row->[2]; + if (!$self->{preserve_case}) { + $tmp{code} =~ s/(TYPE\s+)"[^"]+"\."[^"]+"/$1\L$row->[0]\E/is; + $tmp{code} =~ s/(TYPE\s+)"[^"]+"/$1\L$row->[0]\E/is; + } + push(@types, \%tmp); + } + $sth->finish(); + + $local_dbh->disconnect() if ($local_dbh); + + return \@types; +} + +=head2 _table_info + +This function retrieves all Oracle-native tables information. + +Returns a handle to a DB query statement. + +=cut + +sub _table_info +{ + my $self = shift; + my $do_real_row_count = shift; + + return Ora2Pg::MySQL::_table_info($self) if ($self->{is_mysql}); + + my $owner = ''; + if ($self->{schema}) { + $owner .= "AND A.OWNER='$self->{schema}' "; + } else { + $owner .= "AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + + my %comments = (); + if ($self->{type} eq 'TABLE') + { + my $sql = "SELECT A.TABLE_NAME,A.COMMENTS,A.TABLE_TYPE,A.OWNER FROM ALL_TAB_COMMENTS A, ALL_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='TABLE' $owner"; + if ($self->{db_version} !~ /Release 8/) { + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + } + $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[3].$row->[0]"; + } + $comments{$row->[0]}{comment} = $row->[1]; + $comments{$row->[0]}{table_type} = $row->[2]; + } + $sth->finish(); + } + + my $sql = "SELECT A.OWNER,A.TABLE_NAME,NVL(num_rows,1) NUMBER_ROWS,A.TABLESPACE_NAME,A.NESTED,A.LOGGING,A.PARTITIONED,A.PCT_FREE FROM $self->{prefix}_TABLES A, ALL_OBJECTS O WHERE A.OWNER=O.OWNER AND A.TABLE_NAME=O.OBJECT_NAME AND O.OBJECT_TYPE='TABLE' $owner"; + $sql .= " AND A.TEMPORARY='N' AND (A.NESTED != 'YES' OR A.LOGGING != 'YES') AND A.SECONDARY = 'N'"; + if ($self->{db_version} !~ /Release [89]/) { + $sql .= " AND (A.DROPPED IS NULL OR A.DROPPED = 'NO')"; + } + if ($self->{db_version} !~ /Release 8/) { + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + } + $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + $sql .= " AND (A.IOT_TYPE IS NULL OR A.IOT_TYPE = 'IOT')"; + #$sql .= " ORDER BY A.OWNER, A.TABLE_NAME"; + + my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my %tables_infos = (); + while (my $row = $sth->fetch) + { + if (!$self->{schema} && $self->{export_schema}) { + $row->[1] = "$row->[0].$row->[1]"; + } + $tables_infos{$row->[1]}{owner} = $row->[0] || ''; + $tables_infos{$row->[1]}{num_rows} = $row->[2] || 0; + $tables_infos{$row->[1]}{tablespace} = $row->[3] || 0; + $tables_infos{$row->[1]}{comment} = $comments{$row->[1]}{comment} || ''; + $tables_infos{$row->[1]}{type} = $comments{$row->[1]}{table_type} || ''; + $tables_infos{$row->[1]}{nested} = $row->[4] || ''; + if ($row->[5] eq 'NO') { + $tables_infos{$row->[1]}{nologging} = 1; + } else { + $tables_infos{$row->[1]}{nologging} = 0; + } + if ($row->[6] eq 'NO') { + $tables_infos{$row->[1]}{partitioned} = 0; + } else { + $tables_infos{$row->[1]}{partitioned} = 1; + } + # Only take care of PCTFREE upper than the Oracle default value + if (($row->[7] || 0) > 10) { + $tables_infos{$row->[1]}{fillfactor} = 100 - min(90, $row->[7]); + } + if ($do_real_row_count) + { + $self->logit("DEBUG: looking for real row count for table ($row->[0]) $row->[1] (aka using count(*))...\n", 1); + $sql = "SELECT COUNT(*) FROM $row->[1]"; + if ($self->{schema}) { + $sql = "SELECT COUNT(*) FROM $row->[0].$row->[1]"; + } + my $sth2 = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth2->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my $size = $sth2->fetch(); + $sth2->finish(); + $tables_infos{$row->[1]}{num_rows} = $size->[0]; + } + } + $sth->finish(); + + return %tables_infos; +} + +=head2 _global_temp_table_info + +This function retrieves all Oracle-native global temporary tables information. + +Returns a handle to a DB query statement. + +=cut + +sub _global_temp_table_info +{ + my $self = shift; + + return Ora2Pg::MySQL::_global_temp_table_info($self) if ($self->{is_mysql}); + + my $owner = ''; + if ($self->{schema}) { + $owner .= "AND A.OWNER='$self->{schema}' "; + } else { + $owner .= "AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + + # Get comment on global temporary table + my %comments = (); + if ($self->{type} eq 'TABLE') + { + my $sql = "SELECT A.TABLE_NAME,A.COMMENTS,A.TABLE_TYPE,A.OWNER FROM ALL_TAB_COMMENTS A, ALL_OBJECTS O WHERE A.OWNER=O.OWNER and A.TABLE_NAME=O.OBJECT_NAME and O.OBJECT_TYPE='TABLE' $owner"; + if ($self->{db_version} !~ /Release 8/) { + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)"; + $sql .= " AND (A.OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, TABLE_NAME FROM ALL_OBJECT_TABLES)"; + } + $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[3].$row->[0]"; + } + $comments{$row->[0]}{comment} = $row->[1]; + $comments{$row->[0]}{table_type} = $row->[2]; + } + $sth->finish(); + } + + $sql = "SELECT A.OWNER,A.TABLE_NAME,NVL(num_rows,1) NUMBER_ROWS,A.TABLESPACE_NAME,A.NESTED,A.LOGGING FROM $self->{prefix}_TABLES A, ALL_OBJECTS O WHERE A.OWNER=O.OWNER AND A.TABLE_NAME=O.OBJECT_NAME AND O.OBJECT_TYPE='TABLE' $owner"; + $sql .= " AND A.TEMPORARY='Y'"; + if ($self->{db_version} !~ /Release [89]/) { + $sql .= " AND (A.DROPPED IS NULL OR A.DROPPED = 'NO')"; + } + $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + $sql .= " AND (A.IOT_TYPE IS NULL OR A.IOT_TYPE = 'IOT')"; + #$sql .= " ORDER BY A.OWNER, A.TABLE_NAME"; + + $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my %tables_infos = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[1] = "$row->[0].$row->[1]"; + } + $tables_infos{$row->[1]}{owner} = $row->[0] || ''; + $tables_infos{$row->[1]}{num_rows} = $row->[2] || 0; + $tables_infos{$row->[1]}{tablespace} = $row->[3] || 0; + $tables_infos{$row->[1]}{comment} = $comments{$row->[1]}{comment} || ''; + $tables_infos{$row->[1]}{type} = $comments{$row->[1]}{table_type} || ''; + $tables_infos{$row->[1]}{nested} = $row->[4] || ''; + if ($row->[5] eq 'NO') { + $tables_infos{$row->[1]}{nologging} = 1; + } else { + $tables_infos{$row->[1]}{nologging} = 0; + } + $tables_infos{$row->[1]}{num_rows} = 0; + } + $sth->finish(); + + return %tables_infos; +} + + +=head2 _queries + +This function is used to retrieve all Oracle queries from DBA_AUDIT_TRAIL + +Sets the main hash $self->{queries}. + +=cut + +sub _queries +{ + my ($self) = @_; + + $self->logit("Retrieving audit queries information...\n", 1); + %{$self->{queries}} = $self->_get_audit_queries(); + +} + + +=head2 _get_audit_queries + +This function extract SQL queries from dba_audit_trail + +Returns a hash of queries. + +=cut + +sub _get_audit_queries +{ + my($self) = @_; + + return if (!$self->{audit_user}); + + # If the user is given as not DBA, do not look at tablespace + if ($self->{user_grants}) { + $self->logit("WARNING: Exporting audited queries as non DBA user is not allowed, see USER_GRANT\n", 0); + return; + } + + return Ora2Pg::MySQL::_get_audit_queries($self) if ($self->{is_mysql}); + + my @users = (); + push(@users, split(/[,;\s]/, uc($self->{audit_user}))); + + # Retrieve all object with tablespaces. + my $str = "SELECT SQL_TEXT FROM DBA_AUDIT_TRAIL WHERE ACTION_NAME IN ('INSERT','UPDATE','DELETE','SELECT')"; + if (($#users >= 0) && !grep(/^ALL$/, @users)) { + $str .= " AND USERNAME IN ('" . join("','", @users) . "')"; + } + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %tmp_queries = (); + while (my $row = $sth->fetch) { + $self->_remove_comments(\$row->[0], 1); + $self->{comment_values} = (); + $row->[0] =~ s/\%ORA2PG_COMMENT\d+\%//gs; + $row->[0] = $self->normalize_query($row->[0]); + $tmp_queries{$row->[0]}++; + } + $sth->finish; + + my %queries = (); + my $i = 1; + foreach my $q (keys %tmp_queries) { + $queries{$i} = $q; + $i++; + } + + return %queries; +} + + +=head2 _get_tablespaces + +This function implements an Oracle-native tablespaces information. + +Returns a hash of an array of tablespace names with their system file path. + +=cut + +sub _get_tablespaces +{ + my($self) = @_; + + # If the user is given as not DBA, do not look at tablespace + if ($self->{user_grants}) { + $self->logit("WARNING: Exporting tablespace as non DBA user is not allowed, see USER_GRANT\n", 0); + return; + } + + return Ora2Pg::MySQL::_get_tablespaces($self) if ($self->{is_mysql}); + + # Retrieve all object with tablespaces. +my $str = qq{ +SELECT a.SEGMENT_NAME,a.TABLESPACE_NAME,a.SEGMENT_TYPE,c.FILE_NAME, a.OWNER +FROM DBA_SEGMENTS a, $self->{prefix}_OBJECTS b, DBA_DATA_FILES c +WHERE a.SEGMENT_TYPE IN ('INDEX', 'TABLE', 'INDEX PARTITION', 'TABLE PARTITION') +AND a.SEGMENT_NAME = b.OBJECT_NAME +AND a.SEGMENT_TYPE = b.OBJECT_TYPE +AND a.OWNER = b.OWNER +AND a.TABLESPACE_NAME = c.TABLESPACE_NAME +}; + if ($self->{schema}) { + $str .= " AND a.OWNER='$self->{schema}'"; + } else { + $str .= " AND a.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + $str .= $self->limit_to_objects('TABLESPACE|TABLE', 'a.TABLESPACE_NAME|a.SEGMENT_NAME'); + #$str .= " ORDER BY TABLESPACE_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %tbs = (); + while (my $row = $sth->fetch) { + # TYPE - TABLESPACE_NAME - FILEPATH - OBJECT_NAME + if ($self->{export_schema} && !$self->{schema}) { + $row->[0] = "$row->[4].$row->[0]"; + } + push(@{$tbs{$row->[2]}{$row->[1]}{$row->[3]}}, $row->[0]); + } + $sth->finish; + + return \%tbs; +} + +sub _list_tablespaces +{ + my($self) = @_; + + # If the user is given as not DBA, do not look at tablespace + if ($self->{user_grants}) { + return; + } + + return Ora2Pg::MySQL::_list_tablespaces($self) if ($self->{is_mysql}); + + # list tablespaces. + my $str = qq{ +SELECT c.FILE_NAME, c.TABLESPACE_NAME, a.OWNER, ROUND(c.BYTES/1024000) MB +FROM DBA_DATA_FILES c, DBA_SEGMENTS a +WHERE a.TABLESPACE_NAME = c.TABLESPACE_NAME +}; + if ($self->{schema}) { + $str .= " AND a.OWNER='$self->{schema}'"; + } else { + $str .= " AND a.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + $str .= $self->limit_to_objects('TABLESPACE', 'c.TABLESPACE_NAME'); + #$str .= " ORDER BY c.TABLESPACE_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %tbs = (); + while (my $row = $sth->fetch) { + $tbs{$row->[1]}{path} = $row->[0]; + $tbs{$row->[1]}{owner} = $row->[2]; + } + $sth->finish; + + return \%tbs; +} + + +=head2 _get_partitions + +This function implements an Oracle-native partitions information. +Return two hash ref with partition details and partition default. +=cut + +sub _get_partitions +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_partitions($self) if ($self->{is_mysql}); + + my $highvalue = 'A.HIGH_VALUE'; + if ($self->{db_version} =~ /Release [89]/) { + $highvalue = "'' AS HIGH_VALUE"; + } + my $condition = ''; + if ($self->{schema}) { + $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; + } else { + $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + # Retrieve all partitions. + my $str = qq{ +SELECT + A.TABLE_NAME, + A.PARTITION_POSITION, + A.PARTITION_NAME, + $highvalue, + A.TABLESPACE_NAME, + B.PARTITIONING_TYPE, + C.NAME, + C.COLUMN_NAME, + C.COLUMN_POSITION, + A.TABLE_OWNER +FROM $self->{prefix}_TAB_PARTITIONS A, $self->{prefix}_PART_TABLES B, $self->{prefix}_PART_KEY_COLUMNS C +WHERE + a.table_name = b.table_name AND + (b.partitioning_type = 'RANGE' OR b.partitioning_type = 'LIST' OR b.partitioning_type = 'HASH') + AND a.table_name = c.name + $condition +}; + + if ($self->{db_version} !~ /Release 8/) { + $str .= " AND (A.TABLE_OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)"; + } + $str .= $self->limit_to_objects('TABLE|PARTITION', 'A.TABLE_NAME|A.PARTITION_NAME'); + + if ($self->{prefix} ne 'USER') { + if ($self->{schema}) { + $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; + } else { + $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; + } + } + $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_POSITION,C.COLUMN_POSITION\n"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %parts = (); + my %default = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[9].$row->[0]"; + } + if ( ($row->[3] eq 'DEFAULT')) { + $default{$row->[0]} = $row->[2]; + next; + } + $parts{$row->[0]}{$row->[1]}{name} = $row->[2]; + push(@{$parts{$row->[0]}{$row->[1]}{info}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $row->[7], 'colpos' => $row->[8], 'tablespace' => $row->[4], 'owner' => $row->[9]}); + } + $sth->finish; + + return \%parts, \%default; +} + +=head2 _get_subpartitions + +This function implements an Oracle-native subpartitions information. +Return two hash ref with partition details and partition default. + +=cut + +sub _get_subpartitions +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_subpartitions($self) if ($self->{is_mysql}); + + my $highvalue = 'A.HIGH_VALUE'; + if ($self->{db_version} =~ /Release [89]/) { + $highvalue = "'' AS HIGH_VALUE"; + } + my $condition = ''; + if ($self->{schema}) { + $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; + } else { + $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + # Retrieve all partitions. + my $str = qq{ +SELECT + A.TABLE_NAME, + A.SUBPARTITION_POSITION, + A.SUBPARTITION_NAME, + $highvalue, + A.TABLESPACE_NAME, + B.SUBPARTITIONING_TYPE, + C.NAME, + C.COLUMN_NAME, + C.COLUMN_POSITION, + A.TABLE_OWNER, + A.PARTITION_NAME +FROM $self->{prefix}_tab_subpartitions A, $self->{prefix}_part_tables B, $self->{prefix}_subpart_key_columns C +WHERE + a.table_name = b.table_name AND + (b.subpartitioning_type = 'RANGE' OR b.subpartitioning_type = 'LIST' OR b.subpartitioning_type = 'HASH') + AND a.table_name = c.name + $condition +}; + $str .= $self->limit_to_objects('TABLE|PARTITION', 'A.TABLE_NAME|A.SUBPARTITION_NAME'); + + if ($self->{prefix} ne 'USER') { + if ($self->{schema}) { + $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; + } else { + $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; + } + } + if ($self->{db_version} !~ /Release 8/) { + $str .= " AND (A.TABLE_OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)"; + } + $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_NAME,A.SUBPARTITION_POSITION,C.COLUMN_POSITION\n"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %subparts = (); + my %default = (); + while (my $row = $sth->fetch) { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[9].$row->[0]"; + } + if ( ($row->[3] eq 'MAXVALUE') || ($row->[3] eq 'DEFAULT')) { + $default{$row->[0]}{$row->[10]} = $row->[2]; + next; + } + + $subparts{$row->[0]}{$row->[10]}{$row->[1]}{name} = $row->[2]; + push(@{$subparts{$row->[0]}{$row->[10]}{$row->[1]}{info}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $row->[7], 'colpos' => $row->[8], 'tablespace' => $row->[4], 'owner' => $row->[9]}); + } + $sth->finish; + + return \%subparts, \%default; +} + + +=head2 _synonyms + +This function is used to retrieve all synonyms information. + +Sets the main hash of the synonyms definition $self->{synonyms}. +Keys are the names of all synonyms retrieved from the current +database. + +The synonyms hash is construct as follows: + + $hash{SYNONYM_NAME}{owner} = Owner of the synonym + $hash{SYNONYM_NAME}{table_owner} = Owner of the object referenced by the synonym. + $hash{SYNONYM_NAME}{table_name} = Name of the object referenced by the synonym. + $hash{SYNONYM_NAME}{dblink} = Name of the database link referenced, if any + +=cut + +sub _synonyms +{ + my ($self) = @_; + + # Get all synonyms information + $self->logit("Retrieving synonyms information...\n", 1); + %{$self->{synonyms}} = $self->_get_synonyms(); +} + +=head2 _get_synonyms + +This function implements an Oracle-native synonym information. + +=cut + +sub _get_synonyms +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_synonyms($self) if ($self->{is_mysql}); + + # Retrieve all synonym + my $str = "SELECT OWNER,SYNONYM_NAME,TABLE_OWNER,TABLE_NAME,DB_LINK FROM $self->{prefix}_SYNONYMS"; + if ($self->{schema}) { + $str .= " WHERE (owner='$self->{schema}' OR owner='PUBLIC') AND table_owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } else { + $str .= " WHERE (owner='PUBLIC' OR owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "')) AND table_owner NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + $str .= $self->limit_to_objects('SYNONYM','SYNONYM_NAME'); + #$str .= " ORDER BY SYNONYM_NAME\n"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %synonyms = (); + while (my $row = $sth->fetch) { + next if ($row->[1] =~ /^\//); # Some not fully deleted synonym start with a slash + $synonyms{$row->[1]}{owner} = $row->[0]; + $synonyms{$row->[1]}{table_owner} = $row->[2]; + $synonyms{$row->[1]}{table_name} = $row->[3]; + $synonyms{$row->[1]}{dblink} = $row->[4]; + } + $sth->finish; + + return %synonyms; +} + +=head2 _get_partitions_list + +This function implements an Oracle-native partitions information. +Return a hash of the partition table_name => type +=cut + +sub _get_partitions_list +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_partitions_list($self) if ($self->{is_mysql}); + + my $highvalue = 'A.HIGH_VALUE'; + if ($self->{db_version} =~ /Release [89]/) { + $highvalue = "'' AS HIGH_VALUE"; + } + my $condition = ''; + if ($self->{schema}) { + $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; + } else { + $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + # Retrieve all partitions. + my $str = qq{ +SELECT + A.TABLE_NAME, + A.PARTITION_POSITION, + A.PARTITION_NAME, + $highvalue, + A.TABLESPACE_NAME, + B.PARTITIONING_TYPE, + A.TABLE_OWNER +FROM $self->{prefix}_TAB_PARTITIONS A, $self->{prefix}_PART_TABLES B +WHERE A.TABLE_NAME = B.TABLE_NAME +$condition +}; + if ($self->{db_version} !~ /Release 8/) { + $str .= " AND (A.TABLE_OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)"; + } + $str .= $self->limit_to_objects('TABLE|PARTITION','A.TABLE_NAME|A.PARTITION_NAME'); + + if ($self->{prefix} ne 'USER') { + if ($self->{schema}) { + $str .= "\tAND A.TABLE_OWNER ='$self->{schema}'\n"; + } else { + $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')\n"; + } + } + #$str .= "ORDER BY A.TABLE_NAME\n"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %parts = (); + while (my $row = $sth->fetch) { + $parts{$row->[5]}++; + } + $sth->finish; + + return %parts; +} + +=head2 _get_partitioned_table + +Return a hash of the partitioned table list with the number of partition. + +=cut + +sub _get_partitioned_table +{ + my ($self, %subpart) = @_; + + return Ora2Pg::MySQL::_get_partitioned_table($self) if ($self->{is_mysql}); + + my $highvalue = 'A.HIGH_VALUE'; + if ($self->{db_version} =~ /Release [89]/) { + $highvalue = "'' AS HIGH_VALUE"; + } + my $condition = ''; + if ($self->{schema}) { + $condition .= "AND B.OWNER='$self->{schema}' "; + } else { + $condition .= " AND B.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + # Retrieve all partitions. + my $str = "SELECT B.TABLE_NAME, B.PARTITIONING_TYPE, B.OWNER, B.PARTITION_COUNT, B.SUBPARTITIONING_TYPE"; + if ($self->{type} !~ /SHOW|TEST/) + { + $str .= ", C.COLUMN_NAME, C.COLUMN_POSITION"; + $str .= " FROM $self->{prefix}_PART_TABLES B, $self->{prefix}_PART_KEY_COLUMNS C"; + $str .= " WHERE B.TABLE_NAME = C.NAME AND (B.PARTITIONING_TYPE = 'RANGE' OR B.PARTITIONING_TYPE = 'LIST' OR B.PARTITIONING_TYPE = 'HASH')"; + } + else + { + $str .= " FROM $self->{prefix}_PART_TABLES B WHERE (B.PARTITIONING_TYPE = 'RANGE' OR B.PARTITIONING_TYPE = 'LIST' OR B.PARTITIONING_TYPE = 'HASH') AND B.SUBPARTITIONING_TYPE <> 'SYSTEM' "; + } + $str .= $self->limit_to_objects('TABLE','B.TABLE_NAME'); + + if ($self->{prefix} ne 'USER') + { + if ($self->{type} !~ /SHOW|TEST/) + { + if ($self->{schema}) { + $str .= "\tAND B.OWNER ='$self->{schema}' AND C.OWNER=B.OWNER\n"; + } else { + $str .= "\tAND B.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=C.OWNER\n"; + } + } else { + if ($self->{schema}) { + $str .= "\tAND B.OWNER ='$self->{schema}'\n"; + } else { + $str .= "\tAND B.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')\n"; + } + } + } + if ($self->{db_version} !~ /Release 8/) { + $str .= " AND (B.OWNER, B.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)" if ($self->{type} ne 'FDW'); + } + if ($self->{type} !~ /SHOW|TEST/) { + $str .= "ORDER BY B.OWNER,B.TABLE_NAME,C.COLUMN_POSITION\n"; + } else { + $str .= "ORDER BY B.OWNER,B.TABLE_NAME\n"; + } + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %parts = (); + while (my $row = $sth->fetch) + { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[2].$row->[0]"; + } + # when this is not a composite partition the count is defined + # when this is not the default number of subpartition + $parts{"\L$row->[0]\E"}{count} = 0; + $parts{"\L$row->[0]\E"}{composite} = 0; + if (exists $subpart{"\L$row->[0]\E"}) + { + $parts{"\L$row->[0]\E"}{composite} = 1; + foreach my $k (keys %{$subpart{"\L$row->[0]\E"}}) { + $parts{"\L$row->[0]\E"}{count} += $subpart{"\L$row->[0]\E"}{$k}{count}; + } + $parts{"\L$row->[0]\E"}{count} = $row->[3] if (!$parts{"\L$row->[0]\E"}{count}); + } else { + $parts{"\L$row->[0]\E"}{count} = $row->[3]; + } + $parts{"\L$row->[0]\E"}{type} = $row->[1]; + if ($self->{type} !~ /SHOW|TEST/) { + push(@{ $parts{"\L$row->[0]\E"}{columns} }, $row->[5]); + } + } + $sth->finish; + + return %parts; +} + +=head2 _get_subpartitioned_table + +Return a hash of the partitioned table list with the number of partition. + +=cut + +sub _get_subpartitioned_table +{ + my($self) = @_; + + return Ora2Pg::MySQL::_get_subpartitioned_table($self) if ($self->{is_mysql}); + + my $highvalue = 'A.HIGH_VALUE'; + if ($self->{db_version} =~ /Release [89]/) { + $highvalue = "'' AS HIGH_VALUE"; + } + my $condition = ''; + if ($self->{schema}) { + $condition .= "AND A.TABLE_OWNER='$self->{schema}' "; + } else { + $condition .= " AND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') "; + } + # Retrieve all partitions. + my $str = "SELECT A.TABLE_NAME, A.PARTITION_NAME, A.SUBPARTITION_NAME, A.SUBPARTITION_POSITION, B.SUBPARTITIONING_TYPE, A.TABLE_OWNER, B.PARTITION_COUNT"; + if ($self->{type} !~ /SHOW|TEST/) { + $str .= ", C.COLUMN_NAME, C.COLUMN_POSITION"; + $str .= " FROM $self->{prefix}_TAB_SUBPARTITIONS A, $self->{prefix}_PART_TABLES B, $self->{prefix}_SUBPART_KEY_COLUMNS C"; + } else { + $str .= " FROM $self->{prefix}_TAB_SUBPARTITIONS A, $self->{prefix}_PART_TABLES B"; + } + $str .= " WHERE A.TABLE_NAME = B.TABLE_NAME AND (B.SUBPARTITIONING_TYPE = 'RANGE' OR B.SUBPARTITIONING_TYPE = 'LIST' OR B.SUBPARTITIONING_TYPE = 'HASH')"; + + $str .= " AND A.TABLE_NAME = C.NAME" if ($self->{type} !~ /SHOW|TEST/); + + $str .= $self->limit_to_objects('TABLE|PARTITION','A.TABLE_NAME|A.PARTITION_NAME'); + + if ($self->{prefix} ne 'USER') { + if ($self->{type} !~ /SHOW|TEST/) { + if ($self->{schema}) { + $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; + } else { + $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER AND C.OWNER=A.TABLE_OWNER\n"; + } + } else { + if ($self->{schema}) { + $str .= "\tAND A.TABLE_OWNER ='$self->{schema}' AND B.OWNER=A.TABLE_OWNER\n"; + } else { + $str .= "\tAND A.TABLE_OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') AND B.OWNER=A.TABLE_OWNER\n"; + } + } + } + if ($self->{db_version} !~ /Release 8/) { + $str .= " AND (A.TABLE_OWNER, A.TABLE_NAME) NOT IN (SELECT OWNER, MVIEW_NAME FROM ALL_MVIEWS UNION ALL SELECT LOG_OWNER, LOG_TABLE FROM ALL_MVIEW_LOGS)"; + } + if ($self->{type} !~ /SHOW|TEST/) { + $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_NAME,A.SUBPARTITION_POSITION,C.COLUMN_POSITION\n"; + } else { + $str .= "ORDER BY A.TABLE_OWNER,A.TABLE_NAME,A.PARTITION_NAME,A.SUBPARTITION_POSITION\n"; + } + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %parts = (); + while (my $row = $sth->fetch) + { + if (!$self->{schema} && $self->{export_schema}) { + $row->[0] = "$row->[5].$row->[0]"; + } + $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{type} = $row->[4]; + $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{count}++; + push(@{ $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{columns} }, $row->[7]) if (!grep(/^$row->[7]$/, @{ $parts{"\L$row->[0]\E"}{"\L$row->[1]\E"}{columns} })); + } + $sth->finish; + + return %parts; +} + +sub _get_custom_types +{ + my ($self, $str, $parent) = @_; + + # Copy the type translation hash + my %all_types = %TYPE; + # replace type double precision by single word double + $all_types{'DOUBLE'} = $all_types{'DOUBLE PRECISION'}; + delete $all_types{'DOUBLE PRECISION'}; + # Remove any parenthesis after a type + foreach my $t (keys %all_types) { + $str =~ s/$t\s*\([^\)]+\)/$t/igs; + } + $str =~ s/^[^\(]+\(//s; + $str =~ s/\s*\)\s*;$//s; + $str =~ s/\/\*(.*?)\*\///gs; + $str =~ s/\s*--[^\r\n]+//gs; + my %types_found = (); + my @type_def = split(/\s*,\s*/, $str); + foreach my $s (@type_def) + { + my $cur_type = ''; + if ($s =~ /\s+OF\s+([^\s;]+)/) { + $cur_type = $1; + } elsif ($s =~ /^\s*([^\s]+)\s+([^\s]+)/) { + $cur_type = $2; + } + push(@{$types_found{src_types}}, $cur_type); + if (exists $all_types{$cur_type}) { + push(@{$types_found{pg_types}}, $all_types{$cur_type}); + } + else + { + my $custom_type = $self->_get_types($cur_type); + foreach my $tpe (sort {length($a->{name}) <=> length($b->{name}) } @{$custom_type}) + { + last if (uc($tpe->{name}) eq $cur_type); # prevent infinit loop + $self->logit("\tLooking inside nested custom type $tpe->{name} to extract values...\n", 1); + my %types_def = $self->_get_custom_types($tpe->{code}, $cur_type); + if ($#{$types_def{pg_types}} >= 0) + { + $self->logit("\t\tfound subtype description: $tpe->{name}(" . join(',', @{$types_def{pg_types}}) . ")\n", 1); + push(@{$types_found{pg_types}}, \@{$types_def{pg_types}}); + push(@{$types_found{src_types}}, \@{$types_def{src_types}}); + } + } + } + } + + return %types_found; +} + +sub format_data_row +{ + my ($self, $row, $data_types, $action, $src_data_types, $custom_types, $table, $colcond, $sprep) = @_; + + for (my $idx = 0; $idx <= $#{$data_types}; $idx++) + { + my $data_type = $data_types->[$idx] || ''; + if ($row->[$idx] && $src_data_types->[$idx] =~ /^(SDO_GEOMETRY|ST_|STGEOM_)/) + { + if ($self->{type} ne 'INSERT') + { + if (!$self->{is_mysql} && ($self->{geometry_extract_type} eq 'INTERNAL')) + { + use Ora2Pg::GEOM; + my $geom_obj = new Ora2Pg::GEOM('srid' => $self->{spatial_srid}{$table}->[$idx]); + $geom_obj->{geometry}{srid} = ''; + $row->[$idx] = $geom_obj->parse_sdo_geometry($row->[$idx]); + $row->[$idx] = 'SRID=' . $geom_obj->{geometry}{srid} . ';' . $row->[$idx]; + } + elsif ($self->{geometry_extract_type} eq 'WKB') + { + if ($self->{is_mysql}) { + $row->[$idx] =~ s/^SRID=(\d+);//; + $self->{spatial_srid}{$table}->[$idx] = $1; + } + $row->[$idx] = unpack('H*', $row->[$idx]); + $row->[$idx] = 'SRID=' . $self->{spatial_srid}{$table}->[$idx] . ';' . $row->[$idx]; + } + } + elsif ($self->{geometry_extract_type} eq 'WKB') + { + if ($self->{is_mysql}) + { + $row->[$idx] =~ s/^SRID=(\d+);//; + $self->{spatial_srid}{$table}->[$idx] = $1; + } + $row->[$idx] = unpack('H*', $row->[$idx]); + $row->[$idx] = "'SRID=" . $self->{spatial_srid}{$table}->[$idx] . ';' . $row->[$idx] . "'"; + } + elsif (($self->{geometry_extract_type} eq 'INTERNAL') || ($self->{geometry_extract_type} eq 'WKT')) + { + if (!$self->{is_mysql}) + { + if ($src_data_types->[$idx] =~ /SDO_GEOMETRY/i) + { + use Ora2Pg::GEOM; + my $geom_obj = new Ora2Pg::GEOM('srid' => $self->{spatial_srid}{$table}->[$idx]); + $geom_obj->{geometry}{srid} = ''; + $row->[$idx] = $geom_obj->parse_sdo_geometry($row->[$idx]); + $row->[$idx] = "ST_GeomFromText('" . $row->[$idx] . "', $geom_obj->{geometry}{srid})"; + } + else + { + $row->[$idx] = "ST_Geomtry('" . $row->[$idx] . "', $self->{spatial_srid}{$table}->[$idx])"; + } + } + else + { + $row->[$idx] =~ s/^SRID=(\d+);//; + $row->[$idx] = "ST_GeomFromText('" . $row->[$idx] . "', $1)"; + } + } + } + elsif ($row->[$idx] =~ /^(?!(?!)\x{100})ARRAY\(0x/) + { + print STDERR "/!\\ WARNING /!\\: we should not be there !!!\n"; + } + else + { + + $row->[$idx] = $self->format_data_type($row->[$idx], $data_type, $action, $table, $src_data_types->[$idx], $idx, $colcond->[$idx], $sprep); + + } + } +} + +sub set_custom_type_value +{ + my ($self, $data_type, $user_type, $rows, $dest_type, $no_quote) = @_; + + my $has_array = 0; + my @type_col = (); + my $result = ''; + my $col_ref = []; + push(@$col_ref, @$rows); + my $num_arr = -1; + my $isnested = 0; + + for (my $i = 0; $i <= $#{$col_ref}; $i++) + { + if ($col_ref->[$i] !~ /^ARRAY\(0x/) + { + if ($self->{type} eq 'COPY') + { + # Want to export the user defined type as a single array, not composite type + if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) + { + $has_array = 1; + $col_ref->[$i] =~ s/"/\\\\"/gs; + if ($col_ref->[$i] =~ /[,"]/) { + $col_ref->[$i] = '"' . $col_ref->[$i] . '"'; + }; + # Data must be exported as an array of numeric types + } elsif ($dest_type =~ /\[\d*\]$/) { + $has_array = 1; + } + elsif ($dest_type =~ /(char|text)/) + { + $col_ref->[$i] =~ s/"/\\\\\\\\""/igs; + if ($col_ref->[$i] =~ /[,"]/) { + $col_ref->[$i] = '""' . $col_ref->[$i] . '""'; + }; + } else { + $isnested = 1; + } + } + else + { + # Want to export the user defined type as a single array, not composite type + if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) + { + $has_array = 1; + $col_ref->[$i] =~ s/"/\\"/gs; + $col_ref->[$i] =~ s/'/''/gs; + if ($col_ref->[$i] =~ /[,"]/) { + $col_ref->[$i] = '"' . $col_ref->[$i] . '"'; + }; + # Data must be exported as a simple array of numeric types + } elsif ($dest_type =~ /\[\d*\]$/i) { + $has_array = 1; + } elsif ($dest_type =~ /(char|text)/) { + $col_ref->[$i] = "'" . $col_ref->[$i] . "'" if ($col_ref->[0][$i] ne ''); + } else { + $isnested = 1; + } + } + push(@type_col, $col_ref->[$i]); + } + else + { + $num_arr++; + + my @arr_col = (); + for (my $j = 0; $j <= $#{$col_ref->[$i]}; $j++) + { + # Look for data based on custom type to replace the reference by the value + if ($col_ref->[$i][$j] =~ /^(?!(?!)\x{100})ARRAY\(0x/ + && $user_type->{src_types}[$i][$j] !~ /SDO_GEOMETRY/i + && $user_type->{src_types}[$i][$j] !~ /^(ST_|STGEOM_)/i #ArGis geometry types + ) + { + my $dtype = uc($user_type->{src_types}[$i][$j]) || ''; + $dtype =~ s/\(.*//; # remove any precision + if (!exists $self->{data_type}{$dtype} && !exists $self->{user_type}{$dtype}) { + %{ $self->{user_type}{$dtype} } = $self->custom_type_definition($dtype); + } + $col_ref->[$i][$j] = $self->set_custom_type_value($dtype, $self->{user_type}{$dtype}, $col_ref->[$i][$j], $user_type->{pg_types}[$i][$j], 1); + if ($self->{type} ne 'COPY') { + $col_ref->[$i][$j] =~ s/"/\\\\""/gs; + } else { + $col_ref->[$i][$j] =~ s/"/\\\\\\\\""/gs; + } + } + + if ($self->{type} eq 'COPY') + { + # Want to export the user defined type as charaters array + if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) + { + $has_array = 1; + $col_ref->[$i][$j] =~ s/"/\\\\"/gs; + if ($col_ref->[$i][$j] =~ /[,"]/) { + $col_ref->[$i][$j] = '"' . $col_ref->[$i][$j] . '"'; + }; + } + # Data must be exported as an array of numeric types + elsif ($dest_type =~ /\[\d*\]$/) { + $has_array = 1; + } + } + else + { + # Want to export the user defined type as array + if ($dest_type =~ /(text|char|varying)\[\d*\]$/i) + { + $has_array = 1; + $col_ref->[$i][$j] =~ s/"/\\"/gs; + $col_ref->[$i][$j] =~ s/'/''/gs; + if ($col_ref->[$i][$j] =~ /[,"]/) { + $col_ref->[$i][$j] = '"' . $col_ref->[$i][$j] . '"'; + }; + } + # Data must be exported as an array of numeric types + elsif ($dest_type =~ /\[\d*\]$/) { + $has_array = 1; + } + } + if ($col_ref->[$i][$j] =~ /[\(\)]/ && $col_ref->[$i][$j] !~ /^[\\]+""/) + { + if ($self->{type} ne 'COPY') { + $col_ref->[$i][$j] = "\\\\\"\"" . $col_ref->[$i][$j] . "\\\\\"\""; + } else { + $col_ref->[$i][$j] = "\\\\\\\\\"\"" . $col_ref->[$i][$j] . "\\\\\\\\\"\""; + } + } + push(@arr_col, $col_ref->[$i][$j]); + } + push(@type_col, '(' . join(',', @arr_col) . ')'); + } + } + + if ($has_array) { + $result = '{' . join(',', @type_col) . '}'; + } + elsif ($isnested) + { + # ARRAY[ROW('B','C')] + my $is_string = 0; + foreach my $g (@{$self->{user_type}{$dest_type}->{pg_types}}) { + $is_string = 1 if (grep(/(text|char|varying)/i, @$g)); + } + if ($is_string) { + $result = '({"(' . join(',', @type_col) . ')"})'; + } else { + $result = '("{' . join(',', @type_col) . '}")'; + } + } + else + { + # This is the root call of the function, no global quoting is required + if (!$no_quote) + { + #map { s/^$/NULL/; } @type_col; + #$result = 'ROW(ARRAY[ROW(' . join(',', @type_col) . ')])'; + # With arrays of arrays the construction is different + if ($num_arr > 1) + { + #### Expected + # INSERT: '("{""(0,0,0,0,0,0,0,0,0,,,)"",""(0,0,0,0,0,0,0,0,0,,,)""}")' + # COPY: ("{""(0,0,0,0,0,0,0,0,0,,,)"",""(0,0,0,0,0,0,0,0,0,,,)""}") + #### + $result = "(\"{\"\"" . join('"",""', @type_col) . "\"\"}\")"; + } + # When just one or none arrays are present + else + { + #### Expected + # INSERT: '("(1,1)",0,,)' + # COPY: ("(1,1)",0,,) + #### + map { s/^\(([^\)]+)\)$/"($1)"/; } @type_col; + $result = "(" . join(',', @type_col) . ")"; + } + # else we are in recusive call + } else { + $result = "\"(" . join(',', @type_col) . ")\""; + } + } + if (!$no_quote && $self->{type} ne 'COPY') { + $result = "'$result'"; + } + while ($result =~ s/,"""",/,NULL,/gs) {}; + + return $result; +} + +sub format_data_type +{ + my ($self, $col, $data_type, $action, $table, $src_type, $idx, $cond, $sprep, $isnested) = @_; + + my $q = "'"; + $q = '"' if ($isnested); + + # Skip data type formatting when it has already been done in + # set_custom_type_value(), aka when the data type is an array. + next if ($data_type =~ /\[\d*\]/); + + # Internal timestamp retrieves from custom type is as follow: 01-JAN-77 12.00.00.000000 AM (internal_date_max) + if (($data_type eq 'char') && $col =~ /^(\d{2})-([A-Z]{3})-(\d{2}) (\d{2})\.(\d{2})\.(\d{2}\.\d+) (AM|PM)$/ ) { + my $d = $1; + my $m = $ORACLE_MONTHS{$2}; + my $y = $3; + my $h = $4; + my $min = $5; + my $s = $6; + my $typeh = $7; + if ($typeh eq 'PM') { + $h += 12; + } + if ($d <= $self->{internal_date_max}) { + $d += 2000; + } else { + $d += 1900; + } + $col = "$y-$m-$d $h:$min:$s"; + $data_type = 'timestamp'; + $src_type = 'internal timestamp'; + } + + # Workaround for a bug in DBD::Oracle with the ora_piece_lob option + # (used when no_lob_locator is enabled) where null values fetch as + # empty string for certain types. + if ($self->{no_lob_locator} and ($cond->{clob} or $cond->{blob} or $cond->{long})) { + $col = undef if (!length($col)); + } + + # Preparing data for output + if ($action ne 'COPY') { + if (!defined $col) { + if (!$cond->{isnotnull} || ($self->{empty_lob_null} && ($cond->{clob} || $cond->{isbytea}))) { + $col = 'NULL' if (!$sprep); + } else { + $col = "$q$q"; + } + } elsif ( ($src_type =~ /SDO_GEOMETRY/i) && ($self->{geometry_extract_type} eq 'WKB') ) { + $col = "St_GeomFromWKB($q\\x" . unpack('H*', $col) . "$q, $self->{spatial_srid}{$table}->[$idx])"; + } elsif ($cond->{isbytea}) { + $col = $self->_escape_lob($col, $cond->{raw} ? 'RAW' : 'BLOB', $cond, $isnested); + } elsif ($cond->{istext}) { + if ($cond->{clob}) { + $col = $self->_escape_lob($col, 'CLOB', $cond, $isnested); + } elsif (!$sprep) { + $col = $self->escape_insert($col, $isnested); + } + } elsif ($cond->{isbit}) { + $col = "B$q" . $col . "$q"; + } elsif ($cond->{isdate}) { + if ($col =~ /^0000-00-00/) { + $col = $self->{replace_zero_date} ? "$q$self->{replace_zero_date}$q" : 'NULL'; + } elsif ($col =~ /^(\d+-\d+-\d+ \d+:\d+:\d+)\.$/) { + $col = "$q$1$q"; + } else { + $col = "$q$col$q"; + } + } elsif ($data_type eq 'boolean') { + if (exists $self->{ora_boolean_values}{lc($col)}) { + $col = $q . $self->{ora_boolean_values}{lc($col)} . $q; + } + } else { + $col =~ s/([\-]*)(\~|Inf)/'$1Infinity'/i; + if (!$sprep) { + $col = 'NULL' if ($col eq ''); + } else { + $col = undef if ($col eq ''); + } + } + } else { + if (!defined $col) { + if (!$cond->{isnotnull} || ($self->{empty_lob_null} && ($cond->{clob} || $cond->{isbytea}))) { + $col = '\N'; + } else { + $col = ''; + } + } elsif ( $cond->{geometry} && ($self->{geometry_extract_type} eq 'WKB') ) { + $col = 'SRID=' . $self->{spatial_srid}{$table}->[$idx] . ';' . unpack('H*', $col); + } elsif ($data_type eq 'boolean') { + if (exists $self->{ora_boolean_values}{lc($col)}) { + $col = $self->{ora_boolean_values}{lc($col)}; + } + } elsif ($cond->{isnum}) { + $col =~ s/([\-]*)(\~|Inf)/$1Infinity/i; + $col = '\N' if ($col eq ''); + } elsif ($cond->{isbytea}) { + $col = $self->_escape_lob($col, $cond->{raw} ? 'RAW' : 'BLOB', $cond, $isnested); + } elsif ($cond->{istext}) { + $cond->{clob} ? $col = $self->_escape_lob($col, 'CLOB', $cond, $isnested) : $col = $self->escape_copy($col, $isnested); + } elsif ($cond->{isdate}) { + if ($col =~ /^0000-00-00/) { + $col = $self->{replace_zero_date} || '\N'; + } elsif ($col =~ /^(\d+-\d+-\d+ \d+:\d+:\d+)\.$/) { + $col = $1; + } + } elsif ($cond->{isbit}) { + $col = $col; + } + } + return $col; +} + +sub hs_cond +{ + my ($self, $data_types, $src_data_types, $table) = @_; + + my $col_cond = []; + for (my $idx = 0; $idx < scalar(@$data_types); $idx++) { + my $hs={}; + $hs->{geometry} = $src_data_types->[$idx] =~ /SDO_GEOMETRY/i ? 1 : 0; + $hs->{isnum} = $data_types->[$idx] !~ /^(char|varchar|date|time|text|bytea|xml|uuid|citext)/i ? 1 :0; + $hs->{isdate} = $data_types->[$idx] =~ /^(date|time)/i ? 1 : 0; + $hs->{raw} = $src_data_types->[$idx] =~ /RAW/i ? 1 : 0; + $hs->{clob} = $src_data_types->[$idx] =~ /CLOB/i ? 1 : 0; + $hs->{blob} = $src_data_types->[$idx] =~ /BLOB/i ? 1 : 0; + $hs->{long} = $src_data_types->[$idx] =~ /LONG/i ? 1 : 0; + $hs->{istext} = $data_types->[$idx] =~ /(char|text|xml|uuid|citext)/i ? 1 : 0; + $hs->{isbytea} = $data_types->[$idx] =~ /bytea/i ? 1 : 0; + $hs->{isbit} = $data_types->[$idx] =~ /bit/i ? 1 : 0; + $hs->{isnotnull} = 0; + if ($self->{nullable}{$table}{$idx} =~ /^N/) { + $hs->{isnotnull} = 1; + } + push @$col_cond, $hs; + } + return $col_cond; +} + +sub format_data +{ + my ($self, $rows, $data_types, $action, $src_data_types, $custom_types, $table) = @_; + + my $col_cond = $self->hs_cond($data_types,$src_data_types, $table); + foreach my $row (@$rows) { + $self->format_data_row($row,$data_types,$action,$src_data_types,$custom_types,$table,$col_cond); + } +} + +=head2 dump + +This function dump data to the right export output (gzip file, file or stdout). + +=cut + +sub dump +{ + my ($self, $data, $fh) = @_; + + return if (!defined $data || $data eq ''); + + if (!$self->{compress}) { + if (defined $fh) { + $fh->print($data); + } elsif (defined $self->{fhout}) { + $self->{fhout}->print($data); + } else { + print $data; + } + } elsif ($self->{compress} eq 'Zlib') { + if (not defined $fh) { + $self->{fhout}->gzwrite($data) or $self->logit("FATAL: error dumping compressed data\n", 0, 1); + } else { + $fh->gzwrite($data) or $self->logit("FATAL: error dumping compressed data\n", 0, 1); + } + } elsif (defined $self->{fhout}) { + $self->{fhout}->print($data); + } else { + $self->logit("FATAL: no filehandle to write output, this may not happen\n", 0, 1); + } +} + +=head2 data_dump + +This function dump data to the right output (gzip file, file or stdout) in multiprocess safety. +File is open and locked before writind data, it is closed at end. + +=cut + +sub data_dump +{ + my ($self, $data, $tname, $pname) = @_; + + return if ($self->{oracle_speed}); + + # get out of here if there is no data to dump + return if (not defined $data or $data eq ''); + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + my $filename = $self->{output}; + my $rname = $pname || $tname; + if ($self->{file_per_table}) { + $filename = "${rname}_$self->{output}"; + $filename = "tmp_$filename"; + } + # Set file temporary until the table export is done + $self->logit("Dumping data from $rname to file: $filename\n", 1); + + if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) + { + $self->close_export_file($self->{fhout}) if (defined $self->{fhout} && !$self->{file_per_table} && !$self->{pg_dsn}); + my $fh = $self->append_export_file($filename); + $self->set_binmode($fh) if (!$self->{compress}); + flock($fh, 2) || die "FATAL: can't lock file $dirprefix$filename\n"; + $fh->print($data); + $self->close_export_file($fh); + $self->logit("Written " . length($data) . " bytes to $dirprefix$filename\n", 1); + # Reopen default output file + $self->create_export_file() if (defined $self->{fhout} && !$self->{file_per_table} && !$self->{pg_dsn}); + } + elsif ($self->{file_per_table}) + { + if ($pname) + { + my $fh = $self->append_export_file($filename); + $self->set_binmode($fh) if (!$self->{compress}); + $fh->print($data); + $self->close_export_file($fh); + $self->logit("Written " . length($data) . " bytes to $dirprefix$filename\n", 1); + } + else + { + my $set_encoding = 0; + if (!defined $self->{cfhout}) + { + $self->{cfhout} = $self->open_export_file($filename); + $set_encoding = 1; + } + + if ($self->{compress} eq 'Zlib') + { + $self->{cfhout}->gzwrite($data) or $self->logit("FATAL: error writing compressed data into $filename :: $self->{cfhout}\n", 0, 1); + } + else + { + $self->set_binmode($self->{cfhout}) if (!$self->{compress} && $set_encoding); + $self->{cfhout}->print($data); + } + } + } + else + { + $self->dump($data); + } +} + +=head2 read_config + +This function read the specified configuration file. + +=cut + +sub read_config +{ + my ($self, $file) = @_; + + my $fh = new IO::File; + $fh->open($file) or $self->logit("FATAL: can't read configuration file $file, $!\n", 0, 1); + while (my $l = <$fh>) + { + chomp($l); + $l =~ s/\r//gs; + $l =~ s/^\s*\#.*$//g; + next if (!$l || ($l =~ /^\s+$/)); + $l =~ s/^\s*//; $l =~ s/\s*$//; + my ($var, $val) = split(/\s+/, $l, 2); + $var = uc($var); + if ($var eq 'IMPORT') + { + if ($val) + { + $self->logit("Importing $val...\n", 1); + $self->read_config($val); + $self->logit("Done importing $val.\n",1); + } + } + elsif ($var =~ /^SKIP/) + { + if ($val) + { + $self->logit("No extraction of \L$val\E\n",1); + my @skip = split(/[\s;,]+/, $val); + foreach my $s (@skip) + { + $s = 'indexes' if ($s =~ /^indices$/i); + $AConfig{"skip_\L$s\E"} = 1; + } + } + } + # Should be a else statement but keep the list up to date to memorize the directives full list + elsif (!grep(/^$var$/, 'TABLES','ALLOW','MODIFY_STRUCT','REPLACE_TABLES','REPLACE_COLS', + 'WHERE','EXCLUDE','VIEW_AS_TABLE','ORA_RESERVED_WORDS','SYSUSERS', + 'REPLACE_AS_BOOLEAN','BOOLEAN_VALUES','MODIFY_TYPE','DEFINED_PK', + 'ALLOW_PARTITION','REPLACE_QUERY','FKEY_ADD_UPDATE','DELETE', + 'LOOK_FORWARD_FUNCTION','ORA_INITIAL_COMMAND','PG_INITIAL_COMMAND')) + { + $AConfig{$var} = $val; + if ($var eq 'NO_LOB_LOCATOR') { + print STDERR "WARNING: NO_LOB_LOCATOR is deprecated, use USE_LOB_LOCATOR instead see documentation about the logic change.\n"; + if ($val == 1) { + $AConfig{USE_LOB_LOCATOR} = 0; + } else { + $AConfig{USE_LOB_LOCATOR} = 1; + } + } + if ($var eq 'NO_BLOB_EXPORT') { + print STDERR "WARNING: NO_BLOB_EXPORT is deprecated, use ENABLE_BLOB_EXPORT instead see documentation about the logic change.\n"; + if ($val == 1) { + $AConfig{ENABLE_BLOB_EXPORT} = 0; + } else { + $AConfig{ENABLE_BLOB_EXPORT} = 1; + } + } + } elsif ($var eq 'VIEW_AS_TABLE') { + push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); + } elsif ($var eq 'LOOK_FORWARD_FUNCTION') { + push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); + } + elsif ( ($var eq 'TABLES') || ($var eq 'ALLOW') || ($var eq 'EXCLUDE') + || ($var eq 'ALLOW_PARTITION') ) + { + $var = 'ALLOW' if ($var eq 'TABLES'); + if ($var eq 'ALLOW_PARTITION') + { + $var = 'ALLOW'; + push(@{$AConfig{$var}{PARTITION}}, split(/[,\s]+/, $val) ); + } + else + { + # Syntax: TABLE[regex1 regex2 ...];VIEW[regex1 regex2 ...];glob_regex1 glob_regex2 ... + # Global regex will be applied to the export type only + my @vlist = split(/\s*;\s*/, $val); + foreach my $a (@vlist) + { + if ($a =~ /^([^\[]+)\[(.*)\]$/) { + push(@{$AConfig{$var}{"\U$1\E"}}, split(/[,\s]+/, $2) ); + } else { + push(@{$AConfig{$var}{ALL}}, split(/[,\s]+/, $a) ); + } + } + } + } + elsif ( $var =~ /_INITIAL_COMMAND/ ) { + push(@{$AConfig{$var}}, $val); + } elsif ( $var eq 'SYSUSERS' ) { + push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); + } elsif ( $var eq 'ORA_RESERVED_WORDS' ) { + push(@{$AConfig{$var}}, split(/[\s;,]+/, $val) ); + } + elsif ( $var eq 'FKEY_ADD_UPDATE' ) + { + if (grep(/^$val$/i, @FKEY_OPTIONS)) { + $AConfig{$var} = uc($val); + } else { + $self->logit("FATAL: invalid option, see FKEY_ADD_UPDATE in configuration file\n", 0, 1); + } + } + elsif ($var eq 'MODIFY_STRUCT') + { + while ($val =~ s/([^\(\s]+)\s*\(([^\)]+)\)\s*//) { + my $table = $1; + my $fields = $2; + $fields =~ s/^\s+//; + $fields =~ s/\s+$//; + push(@{$AConfig{$var}{$table}}, split(/[\s,]+/, $fields) ); + } + } + elsif ($var eq 'MODIFY_TYPE') + { + $val =~ s/\\,/#NOSEP#/gs; + my @modif_type = split(/[,;]+/, $val); + foreach my $r (@modif_type) + { + $r =~ s/#NOSEP#/,/gs; + my ($table, $col, $type) = split(/:/, lc($r)); + $AConfig{$var}{$table}{$col} = $type; + } + } + elsif ($var eq 'REPLACE_COLS') + { + while ($val =~ s/([^\(\s]+)\s*\(([^\)]+)\)[,;\s]*//) + { + my $table = $1; + my $fields = $2; + $fields =~ s/^\s+//; + $fields =~ s/\s+$//; + my @rel = split(/[,]+/, $fields); + foreach my $r (@rel) + { + my ($old, $new) = split(/:/, $r); + $AConfig{$var}{$table}{$old} = $new; + } + } + } + elsif ($var eq 'REPLACE_TABLES') + { + my @replace_tables = split(/[\s,;]+/, $val); + foreach my $r (@replace_tables) + { + my ($old, $new) = split(/:/, $r); + $AConfig{$var}{$old} = $new; + } + } + elsif ($var eq 'REPLACE_AS_BOOLEAN') + { + my @replace_boolean = split(/[\s;]+/, $val); + foreach my $r (@replace_boolean) + { + my ($table, $col) = split(/:/, $r); + push(@{$AConfig{$var}{uc($table)}}, uc($col)); + } + } + elsif ($var eq 'BOOLEAN_VALUES') + { + my @replace_boolean = split(/[\s,;]+/, $val); + foreach my $r (@replace_boolean) + { + my ($yes, $no) = split(/:/, $r); + $AConfig{$var}{lc($yes)} = 't'; + $AConfig{$var}{lc($no)} = 'f'; + } + } + elsif ($var eq 'DEFINED_PK') + { + my @defined_pk = split(/[\s,;]+/, $val); + foreach my $r (@defined_pk) + { + my ($table, $col) = split(/:/, lc($r)); + $AConfig{$var}{lc($table)} = $col; + } + } + elsif ($var eq 'WHERE') + { + while ($val =~ s/([^\[\s]+)\s*\[([^\]]+)\]\s*//) + { + my $table = $1; + my $where = $2; + $where =~ s/^\s+//; + $where =~ s/\s+$//; + $AConfig{$var}{$table} = $where; + } + if ($val) { + $AConfig{"GLOBAL_WHERE"} = $val; + } + } + elsif ($var eq 'DELETE') + { + while ($val =~ s/([^\[\s]+)\s*\[([^\]]+)\]\s*//) + { + my $table = $1; + my $delete = $2; + $delete =~ s/^\s+//; + $delete =~ s/\s+$//; + $AConfig{$var}{$table} = $delete; + } + if ($val) { + $AConfig{"GLOBAL_DELETE"} = $val; + } + } + elsif ($var eq 'REPLACE_QUERY') + { + while ($val =~ s/([^\[\s]+)\s*\[([^\]]+)\]\s*//) + { + my $table = lc($1); + my $query = $2; + $query =~ s/^\s+//; + $query =~ s/\s+$//; + $AConfig{$var}{$table} = $query; + } + } + } + $self->close_export_file($fh); + +} + +sub _extract_functions +{ + my ($self, $content) = @_; + + my @lines = split(/\n/s, $content); + my @functions = (''); + my $before = ''; + my $fcname = ''; + my $type = ''; + for (my $i = 0; $i <= $#lines; $i++) { + if ($lines[$i] =~ /^(?:CREATE|CREATE OR REPLACE)?\s*(?:NONEDITIONABLE|EDITIONABLE)?\s*(FUNCTION|PROCEDURE)\s+([a-z0-9_\-\."]+)(.*)/i) { + $type = uc($1); + $fcname = $2; + $fcname =~ s/^.*\.//; + $fcname =~ s/"//g; + $type = 'FUNCTION' if (!$self->{pg_supports_procedure}); + if ($before) { + push(@functions, "$before\n"); + $functions[-1] .= "$type $2 $3\n"; + } else { + push(@functions, "$type $fcname $3\n"); + } + $before = ''; + } elsif ($fcname) { + $functions[-1] .= "$lines[$i]\n"; + } else { + $before .= "$lines[$i]\n"; + } + $fcname = '' if ($lines[$i] =~ /^\s*END\s+$fcname\b/i); + } + + map { s/\bEND\s+(?!IF|LOOP|CASE|INTO|FROM|,)[a-z0-9_]+\s*;/END;/igs; } @functions; + + return @functions; +} + +=head2 _convert_package + +This function is used to rewrite Oracle PACKAGE code to +PostgreSQL SCHEMA. Called only if PLSQL_PGSQL configuration directive +is set to 1. + +=cut + +sub _convert_package +{ + my ($self, $pkg) = @_; + + return if (!$pkg || !exists $self->{packages}{$pkg}); + + my $owner = $self->{packages}{$pkg}{owner} || ''; + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + my $content = ''; + + if ($self->{package_as_schema}) + { + my $pname = $self->quote_object_name($pkg); + $pname =~ s/^[^\.]+\.//; + $content .= "\nDROP SCHEMA $self->{pg_supports_ifexists} $pname CASCADE;\n"; + $content .= "CREATE SCHEMA IF NOT EXISTS $pname;\n"; + if ($self->{force_owner}) + { + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + if ($owner) { + $content .= "ALTER SCHEMA \L$pname\E OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + } + } + # Grab global declaration from the package header + if ($self->{packages}{$pkg}{desc} =~ /CREATE OR REPLACE PACKAGE\s+([^\s]+)(?:\s*\%ORA2PG_COMMENT\d+\%)*\s*(AS|IS)\s*(.*)/is) + { + my $pname = $1; + my $type = $2; + my $glob_declare = $3; + $pname =~ s/"//g; + $pname =~ s/^.*\.//g; + $self->logit("Looking global declaration in package $pname...\n", 1); + + # Process package spec to extract global variables + $self->_remove_comments(\$glob_declare); + if ($glob_declare) + { + my @cursors = (); + ($glob_declare, @cursors) = $self->clear_global_declaration($pname, $glob_declare, 0); + # Then dump custom type + foreach my $tpe (sort {$a->{pos} <=> $b->{pos}} @{$self->{types}}) + { + $self->logit("Dumping type $tpe->{name}...\n", 1); + if ($self->{plsql_pgsql}) { + $tpe->{code} = $self->_convert_type($tpe->{code}, $tpe->{owner}, %{$self->{pkg_type}{$pname}}); + } else { + if ($tpe->{code} !~ /^SUBTYPE\s+/i) { + $tpe->{code} = "CREATE$self->{create_or_replace} $tpe->{code}\n"; + } + } + $tpe->{code} =~ s/REPLACE type/REPLACE TYPE/; + $content .= $tpe->{code} . "\n"; + $i++; + } + $content .= join("\n", @cursors) . "\n"; + $glob_declare = $self->register_global_variable($pname, $glob_declare); + } + @{$self->{types}} = (); + } + + # Convert the package body part + if ($self->{packages}{$pkg}{text} =~ /CREATE OR REPLACE PACKAGE\s+BODY\s*([^\s]+)(?:\s*\%ORA2PG_COMMENT\d+\%)*\s*(AS|IS)\s*(.*)/is) + { + + my $pname = $1; + my $type = $2; + my $ctt = $3; + my $glob_declare = $3; + + $pname =~ s/"//g; + $pname =~ s/^.*\.//g; + $self->logit("Dumping package $pname...\n", 1); + + # Process package spec to extract global variables + $self->_remove_comments(\$glob_declare); + if ($glob_declare && $glob_declare !~ /^(?:\s*\%ORA2PG_COMMENT\d+\%)*(FUNCTION|PROCEDURE)/is) + { + my @cursors = (); + ($glob_declare, @cursors) = $self->clear_global_declaration($pname, $glob_declare, 1); + # Then dump custom type + foreach my $tpe (sort {$a->{pos} <=> $b->{pos}} @{$self->{types}}) + { + next if (!exists $self->{pkg_type}{$pname}{$tpe->{name}}); + $self->logit("Dumping type $tpe->{name}...\n", 1); + if ($self->{plsql_pgsql}) { + $tpe->{code} = $self->_convert_type($tpe->{code}, $tpe->{owner}, %{$self->{pkg_type}{$pname}}); + } else { + if ($tpe->{code} !~ /^SUBTYPE\s+/i) { + $tpe->{code} = "CREATE$self->{create_or_replace} $tpe->{code}\n"; + } + } + $tpe->{code} =~ s/REPLACE type/REPLACE TYPE/; + $content .= $tpe->{code} . "\n"; + $i++; + } + $content .= join("\n", @cursors) . "\n"; + $glob_declare = $self->register_global_variable($pname, $glob_declare); + } + if ($self->{file_per_function}) + { + my $dir = lc("$dirprefix$pname"); + if (!-d "$dir") { + if (not mkdir($dir)) { + $self->logit("Fail creating directory package : $dir - $!\n", 1); + next; + } else { + $self->logit("Creating directory package: $dir\n", 1); + } + } + } + $ctt =~ s/\bEND[^;]*;$//is; + + my @functions = $self->_extract_functions($ctt); + + # Try to detect local function + for (my $i = 0; $i <= $#functions; $i++) + { + my %fct_detail = $self->_lookup_function($functions[$i], $pname); + if (!exists $fct_detail{name}) { + $functions[$i] = ''; + next; + } + $fct_detail{name} =~ s/^.*\.//; + $fct_detail{name} =~ s/"//g; + next if (!$fct_detail{name}); + $fct_detail{name} = lc($fct_detail{name}); + if (!exists $self->{package_functions}{"\L$pname\E"}{$fct_detail{name}}) + { + my $res_name = $fct_detail{name}; + $res_name =~ s/^[^\.]+\.//; + $fct_detail{name} =~ s/^([^\.]+)\.//; + if ($self->{package_as_schema}) { + $res_name = $pname . '.' . $res_name; + } else { + $res_name = $pname . '_' . $res_name; + } + $res_name =~ s/"_"/_/g; + $self->{package_functions}{"\L$pname\E"}{"\L$fct_detail{name}\E"}{name} = $self->quote_object_name($res_name); + $self->{package_functions}{"\L$pname\E"}{"\L$fct_detail{name}\E"}{package} = $pname; + } + } + + $self->{pkgcost} = 0; + foreach my $f (@functions) + { + next if (!$f); + $content .= $self->_convert_function($owner, $f, $pkg || $pname); + } + if ($self->{estimate_cost}) { + $self->{total_pkgcost} += $self->{pkgcost} || 0; + } + + } + + @{$self->{types}} = (); + + return $content; +} + +=head2 _restore_comments + +This function is used to restore comments into SQL code previously +remove for easy parsing + +=cut + +sub _restore_comments +{ + my ($self, $content) = @_; + + # Replace text values that was replaced in code + $self->_restore_text_constant_part($content); + + # Restore comments + while ($$content =~ /(\%ORA2PG_COMMENT\d+\%)[\n]*/is) { + my $id = $1; + my $sep = "\n"; + # Do not append newline if this is a hint + $sep = '' if ($self->{comment_values}{$id} =~ /^\/\*\+/); + $$content =~ s/$id[\n]*/$self->{comment_values}{$id}$sep/is; + delete $self->{comment_values}{$id}; + }; + + # Restore start comment in a constant string + $$content =~ s/\%OPEN_COMMENT\%/\/\*/gs; + + if ($self->{string_constant_regexp}) { + # Replace potential text values that was replaced in comments + $self->_restore_text_constant_part($content); + } +} + +=head2 _remove_comments + +This function is used to remove comments from SQL code +to allow easy parsing + +=cut + +sub _remove_comments +{ + my ($self, $content, $no_constant) = @_; + + # Fix comment in a string constant + while ($$content =~ s/('[^';\n]*)\/\*([^';\n]*')/$1\%OPEN_COMMENT\%$2/s) {}; + + # Fix unterminated comment at end of the code + $$content =~ s/(\/\*(?:(?!\*\/).)*)$/$1 \*\//s; + + # Replace some other cases that are breaking the parser (presence of -- in constant string, etc.) + my @lines = split(/([\n\r]+)/, $$content); + for (my $i = 0; $i <= $#lines; $i++) + { + next if ($lines[$i] !~ /\S/); + + # Single line comment --...-- */ is replaced by */ only + $lines[$i] =~ s/^([\t ]*)\-[\-]+\s*\*\//$1\*\//; + + # Single line comment -- + if ($lines[$i] =~ s/^([\t ]*\-\-.*)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; + $self->{idxcomment}++; + } + + # Single line comment /* ... */ + if ($lines[$i] =~ s/^([\t ]*\/\*.*\*\/)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; + $self->{idxcomment}++; + } + + # ex: v := 'literal' -- commentaire avec un ' guillemet + if ($lines[$i] =~ s/^([^']+'[^']*'\s*)(\-\-.*)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; + $self->{idxcomment}++; + } + + # ex: ---/* REN 16.12.2010 ZKOUSKA TEST NA KOLURC + if ($lines[$i] =~ s/^(\s*)(\-\-(?:(?!\*\/\s*$).)*)$/$1\%ORA2PG_COMMENT$self->{idxcomment}\%/) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $2; + $self->{idxcomment}++; + } + + # ex: var1 := SUBSTR(var2,1,28) || ' -- ' || var3 || ' -- ' || SUBSTR(var4,1,26) ; + while ($lines[$i] =~ s/('[^;']*\-\-[^']*')/\?TEXTVALUE$self->{text_values_pos}\?/) + { + $self->{text_values}{$self->{text_values_pos}} = $1; + $self->{text_values_pos}++; + } + } + $$content =join('', @lines); + + # First remove hints they are not supported in PostgreSQL and it break the parser + while ($$content =~ s/(\/\*\+(?:.*?)\*\/)/\%ORA2PG_COMMENT$self->{idxcomment}\%/s) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; + $self->{idxcomment}++; + } + + # Replace /* */ comments by a placeholder and save the comment + while ($$content =~ s/(\/\*(.*?)\*\/)/\%ORA2PG_COMMENT$self->{idxcomment}\%/s) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; + $self->{idxcomment}++; + } + + while ($$content =~ s/(\'[^\'\n\r]+\b(PROCEDURE|FUNCTION)\s+[^\'\n\r]+\')/\%ORA2PG_COMMENT$self->{idxcomment}\%/is) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; + $self->{idxcomment}++; + } + @lines = split(/\n/, $$content); + for (my $j = 0; $j <= $#lines; $j++) + { + if (!$self->{is_mysql}) + { + # Extract multiline comments as a single placeholder + my $old_j = $j; + my $cmt = ''; + while ($lines[$j] =~ /^(\s*\-\-.*)$/) + { + $cmt .= "$1\n"; + $j++; + } + if ( $j > $old_j ) + { + chomp($cmt); + $lines[$old_j] =~ s/^(\s*\-\-.*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/; + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $cmt; + $self->{idxcomment}++; + $j--; + while ($j > $old_j) + { + delete $lines[$j]; + $j--; + } + } + my $nocomment = ''; + if ($lines[$j] =~ s/^([^']*)('[^\-\']*\-\-[^\-\']*')/$1\%NO_COMMENT\%/) { + $nocomment = $2; + } + if ($lines[$j] =~ s/(\s*\-\-.*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; + chomp($self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"}); + $self->{idxcomment}++; + } + $lines[$j] =~ s/\%NO_COMMENT\%/$nocomment/; + } + else + { + # Mysql supports differents kinds of comment's starter + if ( ($lines[$j] =~ s/(\s*\-\- .*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) || + (!grep(/^$self->{type}$/, 'FUNCTION', 'PROCEDURE') && $lines[$j] =~ s/(\s*COMMENT\s+'.*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) || + ($lines[$j] =~ s/(\s*\# .*)$/\%ORA2PG_COMMENT$self->{idxcomment}\%/) ) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; + chomp($self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"}); + # Normalize start of comment + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} =~ s/^(\s*)COMMENT/$1\-\- /; + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} =~ s/^(\s*)\#/$1\-\- /; + $self->{idxcomment}++; + } + } + } + $$content = join("\n", @lines); + + # Replace subsequent comment by a single one + while ($$content =~ s/(\%ORA2PG_COMMENT\d+\%\s*\%ORA2PG_COMMENT\d+\%)/\%ORA2PG_COMMENT$self->{idxcomment}\%/s) + { + $self->{comment_values}{"\%ORA2PG_COMMENT$self->{idxcomment}\%"} = $1; + $self->{idxcomment}++; + } + + # Restore possible false positive constant replacement inside comment + foreach my $k (keys %{ $self->{comment_values} } ) { + $self->{comment_values}{$k} =~ s/\?TEXTVALUE(\d+)\?/$self->{text_values}{$1}/gs; + } + + # Then replace text constant part to prevent a split on a ; or -- inside a text + if (!$no_constant) { + $self->_remove_text_constant_part($content); + } +} + +=head2 _convert_function + +This function is used to rewrite Oracle FUNCTION code to +PostgreSQL. Called only if PLSQL_PGSQL configuration directive +is set to 1. + +=cut + +sub _convert_function +{ + my ($self, $owner, $plsql, $pname) = @_; + + my $dirprefix = ''; + $dirprefix = "$self->{output_dir}/" if ($self->{output_dir}); + + my %fct_detail = $self->_lookup_function($plsql, $pname); + if ($self->{is_mysql}) { + $pname = ''; + } + return if (!exists $fct_detail{name}); + + $fct_detail{name} =~ s/^.*\.//; + $fct_detail{name} =~ s/"//gs; + + my $sep = '.'; + $sep = '_' if (!$self->{package_as_schema}); + my $fname = $self->quote_object_name($fct_detail{name}); + $fname = $self->quote_object_name("$pname$sep$fct_detail{name}") if ($pname && !$self->{is_mysql}); + $fname =~ s/"_"/_/gs; + + $fct_detail{args} =~ s/\s+IN\s+/ /igs; # Remove default IN keyword + # Replace DEFAULT EMPTY_BLOB() from function/procedure arguments by DEFAULT NULL + $fct_detail{args} =~ s/\s+DEFAULT\s+EMPTY_[CB]LOB\(\)/DEFAULT NULL/igs; + + # Input parameters after one with a default value must also have defaults + # we add DEFAULT NULL to all remaining parameter without a default value. + my @args_sorted = (); + $fct_detail{args} =~ s/^\((.*)\)(\s*\%ORA2PG_COMMENT\d+\%)*\s*$/$1$2/gs; + if ($self->{use_default_null}) + { + my $has_default = 0; + @args_sorted = split(',', $fct_detail{args}); + for (my $i = 0; $i <= $#args_sorted; $i++) + { + $has_default = 1 if ($args_sorted[$i] =~ /\s+DEFAULT\s/i); + if ($has_default && $args_sorted[$i] !~ /\s+DEFAULT\s/i) { + $args_sorted[$i] .= ' DEFAULT NULL'; + } + } + } + else + { + # or we need to sort the arguments so the ones with default values will be on the bottom + push(@args_sorted, grep {!/\sdefault\s/i} split ',', $fct_detail{args}); + push(@args_sorted, grep {/\sdefault\s/i} split ',', $fct_detail{args}); + my @orig_args = split(',', $fct_detail{args}); + + # Show a warning when there is parameters reordering + my $fct_warning = ''; + for (my $i = 0; $i <= $#args_sorted; $i++) + { + if ($args_sorted[$i] ne $orig_args[$i]) + { + my $str = $fct_detail{args}; + $str =~ s/\%ORA2PG_COMMENT\d+\%//sg; + $str =~ s/[\n\r]+//gs; + $str =~ s/\s+/ /g; + $self->_restore_text_constant_part(\$str); + $fct_warning = "\n-- WARNING: parameters order has been changed by Ora2Pg to move parameters with default values at end\n"; + $fct_warning .= "-- Original order was: $fname($str)\n"; + $fct_warning .= "-- You will need to manually reorder parameters in the function calls\n"; + print STDERR $fct_warning; + last; + } + } + } + + # Apply parameter list with translation for default values and reordering if needed + for (my $i = 0; $i <= $#args_sorted; $i++) + { + if ($args_sorted[$i] =~ / DEFAULT ([^'].*)/i) + { + my $cod = Ora2Pg::PLSQL::convert_plsql_code($self, $1); + $args_sorted[$i] =~ s/( DEFAULT )([^'].*)/$1$cod/i; + } + } + $fct_detail{args} = '(' . join(',', @args_sorted) . ')'; + + # Set the return part + my $func_return = ''; + $fct_detail{setof} = ' SETOF' if ($fct_detail{setof}); + + my $search_path = ''; + if ($self->{export_schema} && !$self->{schema}) { + $search_path = $self->set_search_path($owner); + } + + # PostgreSQL procedure do not support OUT parameter, translate them into INOUT params + if ($self->{pg_supports_procedure} && ($fct_detail{args} =~ /\bOUT\s+[^,\)]+/i)) { + $fct_detail{args} =~ s/\bOUT(\s+[^,\)]+)/INOUT$1/igs; + } + + my @nout = $fct_detail{args} =~ /\bOUT\s+([^,\)]+)/igs; + my @ninout = $fct_detail{args} =~ /\bINOUT\s+([^,\)]+)/igs; + if ($fct_detail{hasreturn}) + { + my $nbout = $#nout+1 + $#ninout+1; + # When there is one or more out parameter, let PostgreSQL + # choose the right type with not using a RETURNS clause. + if ($nbout > 0) { + $func_return = " AS \$body\$\n"; + } else { + # Returns the right type + $func_return = " RETURNS$fct_detail{setof} $fct_detail{func_ret_type} AS \$body\$\n"; + } + } + elsif (!$self->{pg_supports_procedure}) + { + # Return void when there's no out parameters + if (($#nout < 0) && ($#ninout < 0)) { + $func_return = " RETURNS VOID AS \$body\$\n"; + } else { + # When there is one or more out parameter, let PostgreSQL + # choose the right type with not using a RETURNS clause. + $func_return = " AS \$body\$\n"; + } + } + else + { + $func_return = " AS \$body\$\n"; + } + + # extract custom type declared in a stored procedure + my $create_type = ''; + while ($fct_detail{declare} =~ s/\s+TYPE\s+([^\s]+)\s+IS\s+RECORD\s*\(([^;]+)\)\s*;//is) + { + $create_type .= "CREATE TYPE $1 AS ($2);\n"; + } + + my @at_ret_param = (); + my @at_ret_type = (); + my $at_suffix = ''; + my $at_inout = 0; + if ($fct_detail{declare} =~ s/\s*(PRAGMA\s+AUTONOMOUS_TRANSACTION[\s;]*)/-- $1/is && $self->{autonomous_transaction}) + { + $at_suffix = '_atx'; + # COMMIT is not allowed in PLPGSQL function + $fct_detail{code} =~ s/\bCOMMIT\s*;//; + # Remove the pragma when a conversion is done + $fct_detail{declare} =~ s/--\s+PRAGMA\s+AUTONOMOUS_TRANSACTION[\s;]*//is; + my @tmp = split(',', $fct_detail{args}); + $tmp[0] =~ s/^\(//; + $tmp[-1] =~ s/\)$//; + foreach my $p (@tmp) + { + if ($p =~ s/\bOUT\s+//) + { + $at_inout++; + push(@at_ret_param, $p); + push(@at_ret_type, $p); + } + elsif ($p =~ s/\bINOUT\s+//) + { + $at_inout++; + push(@at_ret_param, $p); + push(@at_ret_type, $p); + } + } + map { s/^(.*?) //; } @at_ret_type; + if ($fct_detail{hasreturn} && $#at_ret_param < 0) + { + push(@at_ret_param, 'ret ' . $fct_detail{func_ret_type}); + push(@at_ret_type, $fct_detail{func_ret_type}); + } + map { s/^\s+//; } @at_ret_param; + map { s/\s+$//; } @at_ret_param; + map { s/^\s+//; } @at_ret_type; + map { s/\s+$//; } @at_ret_type; + } + + my $name = $fname; + my $type = $fct_detail{type}; + $type = 'FUNCTION' if (!$self->{pg_supports_procedure}); + + my $function = "\n$create_type\n\n${fct_warning}CREATE$self->{create_or_replace} $type $fname$at_suffix $fct_detail{args}"; + if (!$pname || !$self->{package_as_schema}) + { + if ($self->{export_schema} && !$self->{schema}) + { + $function = "\n${fct_warning}CREATE$self->{create_or_replace} $type " . $self->quote_object_name("$owner.$fname") . " $fct_detail{args}"; + $name = $self->quote_object_name("$owner.$fname"); + $self->logit("Parsing function " . $self->quote_object_name("$owner.$fname") . "...\n", 1); + } + elsif ($self->{export_schema} && $self->{schema}) + { + $function = "\n${fct_warning}CREATE$self->{create_or_replace} $type " . $self->quote_object_name("$self->{schema}.$fname") . " $fct_detail{args}"; + $name = $self->quote_object_name("$self->{schema}.$fname"); + $self->logit("Parsing function " . $self->quote_object_name("$self->{schema}.$fname") . "...\n", 1); + } + } + else + { + $self->logit("Parsing function $fname...\n", 1); + } + + # Create a wrapper for the function if we found an autonomous transaction + my $at_wrapper = ''; + if ($at_suffix && !$self->{pg_background}) + { + $at_wrapper = qq{ +$search_path +-- +-- dblink wrapper to call function $name as an autonomous transaction +-- +CREATE EXTENSION IF NOT EXISTS dblink; + +}; + $at_wrapper .= "CREATE$self->{create_or_replace} $type $name $fct_detail{args}$func_return"; + my $params = ''; + if ($#{$fct_detail{at_args}} >= 0) + { + map { s/(.+)/quote_nullable($1)/; } @{$fct_detail{at_args}}; + $params = " ' || " . join(" || ',' || ", @{$fct_detail{at_args}}) . " || ' "; + } + my $dblink_conn = $self->{dblink_conn} || "'port=5432 dbname=testdb host=localhost user=pguser password=pgpass'"; + $at_wrapper .= qq{DECLARE + -- Change this to reflect the dblink connection string + v_conn_str text := $dblink_conn; + v_query text; +}; + if ($#at_ret_param == 0) + { + my $varname = $at_ret_param[0]; + $varname =~ s/\s+.*//; + my $vartype = $at_ret_type[0]; + $vartype =~ s/.*\s+//; + if (!$fct_detail{hasreturn}) + { + $at_wrapper .= qq{ +BEGIN + v_query := 'SELECT * FROM $fname$at_suffix ($params)'; + SELECT v_ret INTO $varname FROM dblink(v_conn_str, v_query) AS p (v_ret $vartype); +}; + } + else + { + $at_ret_type[0] = $fct_detail{func_ret_type}; + $at_ret_param[0] = 'ret ' . $fct_detail{func_ret_type}; + $at_wrapper .= qq{ + v_ret $at_ret_type[0]; +BEGIN + v_query := 'SELECT * FROM $fname$at_suffix ($params)'; + SELECT * INTO v_ret FROM dblink(v_conn_str, v_query) AS p ($at_ret_param[0]); + RETURN v_ret; +}; + } + } + elsif ($#at_ret_param > 0) + { + my $varnames = ''; + my $vartypes = ''; + for (my $i = 0; $i <= $#at_ret_param; $i++) + { + my $v = $at_ret_param[$i]; + $v =~ s/\s+.*//; + $varnames .= "$v, "; + $vartypes .= "v_ret$i "; + my $t = $at_ret_type[$i]; + $t =~ s/.*\s+//; + $vartypes .= "$t, "; + } + $varnames =~ s/, $//; + $vartypes =~ s/, $//; + if (!$fct_detail{hasreturn}) + { + $at_wrapper .= qq{ +BEGIN + v_query := 'SELECT * FROM $fname$at_suffix ($params)'; + SELECT * FROM dblink(v_conn_str, v_query) AS p ($vartypes) INTO $varnames; +}; + } + else + { + $at_ret_type[0] = $fct_detail{func_ret_type}; + $at_ret_param[0] = 'ret ' . $fct_detail{func_ret_type}; + $at_wrapper .= qq{ + v_ret $at_ret_type[0]; +BEGIN + v_query := 'SELECT * FROM $fname$at_suffix ($params)'; + SELECT * INTO v_ret FROM dblink(v_conn_str, v_query) AS p ($at_ret_param[0]); + RETURN v_ret; +}; + } + } + elsif (!$fct_detail{hasreturn}) + { + $at_wrapper .= qq{ +BEGIN + v_query := 'SELECT true FROM $fname$at_suffix ($params)'; + PERFORM * FROM dblink(v_conn_str, v_query) AS p (ret boolean); +}; + } + else + { + print STDERR "WARNING: we should not be there, please send the Oracle code of the $self->{type} to the author for debuging.\n"; + } + $at_wrapper .= qq{ +END; +\$body\$ LANGUAGE plpgsql SECURITY DEFINER; +}; + + } + elsif ($at_suffix && $self->{pg_background}) + { + $at_wrapper = qq{ +$search_path +-- +-- pg_background wrapper to call function $name as an autonomous transaction +-- +CREATE EXTENSION IF NOT EXISTS pg_background; + +}; + $at_wrapper .= "CREATE$self->{create_or_replace} $type $name $fct_detail{args}$func_return"; + my $params = ''; + if ($#{$fct_detail{at_args}} >= 0) + { + map { s/(.+)/quote_nullable($1)/; } @{$fct_detail{at_args}}; + $params = " ' || " . join(" || ',' || ", @{$fct_detail{at_args}}) . " || ' "; + } + + $at_wrapper .= qq{ +DECLARE + v_query text; +}; + if (!$fct_detail{hasreturn}) + { + $at_wrapper .= qq{ +BEGIN + v_query := 'SELECT true FROM $fname$at_suffix ($params)'; + PERFORM * FROM pg_background_result(pg_background_launch(v_query)) AS p (ret boolean); +}; + } + elsif ($#at_ret_param == 0) + { + my $prm = join(',', @at_ret_param); + $at_wrapper .= qq{ + v_ret $at_ret_type[0]; +BEGIN + v_query := 'SELECT * FROM $fname$at_suffix ($params)'; + SELECT * INTO v_ret FROM pg_background_result(pg_background_launch(v_query)) AS p ($at_ret_param[0]); + RETURN v_ret; +}; + } + $at_wrapper .= qq{ +END; +\$body\$ LANGUAGE plpgsql SECURITY DEFINER; +}; + + + } + + # Add the return part of the function declaration + $function .= $func_return; + if ($fct_detail{immutable}) { + $fct_detail{immutable} = ' IMMUTABLE'; + } elsif ($plsql =~ /^FUNCTION/i) + { + # Oracle function can't modify data so always mark them as stable + if ($self->{function_stable}) { + $fct_detail{immutable} = ' STABLE'; + } + } + if ($language && ($language !~ /SQL/i)) { + $function .= "AS '$fct_detail{library}', '$fct_detail{library_fct}'\nLANGUAGE $language$fct_detail{immutable};\n"; + $function =~ s/AS \$body\$//; + } + + my $revoke = ''; + if ($fct_detail{code}) + { + $fct_detail{declare} = '' if ($fct_detail{declare} !~ /[a-z]/is); + $fct_detail{declare} =~ s/^\s*DECLARE//i; + $fct_detail{declare} .= ';' if ($fct_detail{declare} && $fct_detail{declare} !~ /;\s*$/s && $fct_detail{declare} !~ /\%ORA2PG_COMMENT\d+\%\s*$/s); + my $code_part = ''; + $code_part .= "DECLARE\n$fct_detail{declare}\n" if ($fct_detail{declare}); + $fct_detail{code} =~ s/^BEGIN\b//is; + $code_part .= "BEGIN" . $fct_detail{code}; + # Replace PL/SQL code into PL/PGSQL similar code + $function .= Ora2Pg::PLSQL::convert_plsql_code($self, $code_part); + $function .= ';' if ($function !~ /END\s*;\s*$/is && $fct_detail{code} !~ /\%ORA2PG_COMMENT\d+\%\s*$/); + $function .= "\n\$body\$\nLANGUAGE PLPGSQL\n"; + + # Remove parameters to RETURN call when the function has no RETURNS clause + if ($function !~ /\s+RETURNS\s+/s || ($function =~ /\s+RETURNS VOID\s+/s || ($type eq 'PROCEDURE' && $self->{pg_supports_procedures}))) { + $self->_remove_text_constant_part(\$function); + $function =~ s/(RETURN)\s+[^;]+;/$1;/igs; + $self->_restore_text_constant_part(\$function); + } + $revoke = "-- REVOKE ALL ON $type $name $fct_detail{args} FROM PUBLIC;"; + $revoke =~ s/[\n\r]+\s*/ /gs; + $revoke .= "\n"; + if ($self->{force_security_invoker}) { + $function .= "SECURITY INVOKER\n"; + } + else + { + if ($self->{type} ne 'PACKAGE') + { + if (!$self->{is_mysql}) { + $function .= "SECURITY DEFINER\n" if ($self->{security}{"\U$fct_detail{name}\E"}{security} eq 'DEFINER'); + } else { + $function .= "SECURITY DEFINER\n" if ($fct_detail{security} eq 'DEFINER'); + } + } + else + { + $function .= "SECURITY DEFINER\n" if ($self->{security}{"\U$pname\E"}{security} eq 'DEFINER'); + } + } + $fct_detail{immutable} = '' if ($fct_detail{code} =~ /\b(UPDATE|INSERT|DELETE)\b/is); + $function .= "$fct_detail{immutable};\n"; + $function = "\n$fct_detail{before}$function"; + } + + if ($self->{force_owner}) { + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + if ($owner) { + $function .= "ALTER $type $fname $fct_detail{args} OWNER TO"; + $function .= " " . $self->quote_object_name($owner) . ";\n"; + } + } + $function .= "\nCOMMENT ON FUNCTION $fname$at_suffix $fct_detail{args} IS $fct_detail{comment};\n" if ($fct_detail{comment}); + $function .= $revoke; + $function = $at_wrapper . $function; + + $fname =~ s/"//g; # Remove case sensitivity quoting + $fname =~ s/^$pname\.//i; # remove package name + if ($pname && $self->{file_per_function}) { + $self->logit("\tDumping to one file per function: $dirprefix\L$pname/$fname\E_$self->{output}\n", 1); + my $sql_header = "-- Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION\n"; + $sql_header .= "-- Copyright 2000-2020 Gilles DAROLD. All rights reserved.\n"; + $sql_header .= "-- DATASOURCE: $self->{oracle_dsn}\n\n"; + if ($self->{client_encoding}) { + $sql_header .= "SET client_encoding TO '\U$self->{client_encoding}\E';\n"; + } + $sql_header .= $self->set_search_path(); + $sql_header .= "SET check_function_bodies = false;\n\n" if (!$self->{function_check}); + $sql_header = '' if ($self->{no_header}); + + my $fhdl = $self->open_export_file("$dirprefix\L$pname/$fname\E_$self->{output}", 1); + $self->set_binmode($fhdl) if (!$self->{compress}); + $self->_restore_comments(\$function); + $self->normalize_function_call(\$function); + $function =~ s/(-- REVOKE ALL ON (?:FUNCTION|PROCEDURE) [^;]+ FROM PUBLIC;)/&remove_newline($1)/sge; + $self->dump($sql_header . $function, $fhdl); + $self->close_export_file($fhdl); + my $f = "$dirprefix\L$pname/$fname\E_$self->{output}"; + $f =~ s/\.(?:gz|bz2)$//i; + $function = "\\i$self->{psql_relative_path} $f\n"; + $self->save_filetoupdate_list(lc($pname), lc($fname), "$dirprefix\L$pname/$fname\E_$self->{output}"); + return $function; + } elsif ($pname) { + $self->save_filetoupdate_list(lc($pname), lc($fname), "$dirprefix$self->{output}"); + } + + $function =~ s/\r//gs; + my @lines = split(/\n/, $function); + map { s/^\/$//; } @lines; + + return join("\n", @lines); +} + +=head2 _convert_declare + +This function is used to rewrite Oracle FUNCTION declaration code +to PostgreSQL. Called only if PLSQL_PGSQL configuration directive +is set to 1. + +=cut + +sub _convert_declare +{ + my ($self, $declare) = @_; + + $declare =~ s/\s+$//s; + + return if (!$declare); + + my @allwithcomments = split(/(\%ORA2PG_COMMENT\d+\%\n*)/s, $declare); + for (my $i = 0; $i <= $#allwithcomments; $i++) { + next if ($allwithcomments[$i] =~ /ORA2PG_COMMENT/); + my @pg_declare = (); + foreach my $tmp_var (split(/;/,$allwithcomments[$i])) { + # Not cursor declaration + if ($tmp_var !~ /\bcursor\b/is) { + # Extract default assignment + my $tmp_assign = ''; + if ($tmp_var =~ s/\s*(:=|DEFAULT)(.*)$//is) { + $tmp_assign = " $1$2"; + } + # Extract variable name and type + my $tmp_pref = ''; + my $tmp_name = ''; + my $tmp_type = ''; + if ($tmp_var =~ /(\s*)([^\s]+)\s+(.*?)$/s) { + $tmp_pref = $1; + $tmp_name = $2; + $tmp_type = $3; + $tmp_type =~ s/\s+//gs; + if ($tmp_type =~ /([^\(]+)\(([^\)]+)\)/) { + my $type_name = $1; + my ($prec, $scale) = split(/,/, $2); + $scale ||= 0; + my $len = $prec; + $prec = 0 if (!$scale); + $len =~ s/\D//g; + $tmp_type = $self->_sql_type($type_name,$len,$prec,$scale,$tmp_assign); + } else { + $tmp_type = $self->_sql_type($tmp_type); + } + push(@pg_declare, "$tmp_pref$tmp_name $tmp_type$tmp_assign;"); + } + } else { + push(@pg_declare, "$tmp_var;"); + } + } + $allwithcomments[$i] = join("", @pg_declare); + } + + return join("", @allwithcomments); +} + + +=head2 _format_view + +This function is used to rewrite Oracle VIEW declaration code +to PostgreSQL. + +=cut + +sub _format_view +{ + my ($self, $view, $sqlstr) = @_; + + $self->_remove_comments(\$sqlstr); + + # Retrieve the column part of the view to remove double quotes + if (!$self->{preserve_case} && $sqlstr =~ s/^(.*?)\bFROM\b/FROM/is) { + my $tmp = $1; + $tmp =~ s/"//gs; + $sqlstr = $tmp . $sqlstr; + } + + my @tbs = (); + # Retrieve all tbs names used in view if possible + if ($sqlstr =~ /\bFROM\b(.*)/is) { + my $tmp = $1; + $tmp =~ s/\%ORA2PG_COMMENT\d+\%//gs; + $tmp =~ s/\s+/ /gs; + $tmp =~ s/\bWHERE.*//is; + # Remove all SQL reserved words of FROM STATEMENT + $tmp =~ s/(LEFT|RIGHT|INNER|OUTER|NATURAL|CROSS|JOIN|\(|\))//igs; + # Remove all ON join, if any + $tmp =~ s/\bON\b[A-Z_\.\s]*=[A-Z_\.\s]*//igs; + # Sub , with whitespace + $tmp =~ s/,/ /g; + my @tmp_tbs = split(/\s+/, $tmp); + foreach my $p (@tmp_tbs) { + push(@tbs, $p) if ($p =~ /^[A-Z_0-9\$]+$/i); + } + } + foreach my $tb (@tbs) { + next if (!$tb); + my $regextb = $tb; + $regextb =~ s/\$/\\\$/g; + if (!$self->{preserve_case}) { + # Escape column name + $sqlstr =~ s/["']*\b$regextb\b["']*\.["']*([A-Z_0-9\$]+)["']*(,?)/$tb.$1$2/igs; + # Escape table name + $sqlstr =~ s/(^=\s?)["']*\b$regextb\b["']*/$tb/igs; + } else { + # Escape column name + $sqlstr =~ s/["']*\b${regextb}["']*\.["']*([A-Z_0-9\$]+)["']*(,?)/"$tb"."$1"$2/igs; + # Escape table name + $sqlstr =~ s/(^=\s?)["']*\b$regextb\b["']*/"$tb"/igs; + if ($tb =~ /(.*)\.(.*)/) { + my $prefx = $1; + my $sufx = $2; + $sqlstr =~ s/"$regextb"/"$prefx"\."$sufx/g; + } + } + } + + # replace column name in view query definition if needed + foreach my $c (sort { $b cmp $a } keys %{ $self->{replaced_cols}{"\L$view\E"} }) + { + my $nm = $self->{replaced_cols}{"\L$view\E"}{$c}; + $sqlstr =~ s/([\(,\s\."])$c([,\s\.:"\)])/$1$nm$2/ig; + } + + if ($self->{plsql_pgsql}) { + $sqlstr = Ora2Pg::PLSQL::convert_plsql_code($self, $sqlstr); + } + + $self->_restore_comments(\$sqlstr); + + return $sqlstr; +} + +=head2 randpattern + +This function is used to replace the use of perl module String::Random +and is simply a cut & paste from this module. + +=cut + +sub randpattern +{ + my $patt = shift; + + my $string = ''; + + my @upper=("A".."Z"); + my @lower=("a".."z"); + my @digit=("0".."9"); + my %patterns = ( + 'C' => [ @upper ], + 'c' => [ @lower ], + 'n' => [ @digit ], + ); + for my $ch (split(//, $patt)) { + if (exists $patterns{$ch}) { + $string .= $patterns{$ch}->[int(rand(scalar(@{$patterns{$ch}})))]; + } else { + $string .= $ch; + } + } + + return $string; +} + +=head2 logit + +This function log information to STDOUT or to a logfile +following a debug level. If critical is set, it dies after +writing to log. + +=cut + +sub logit +{ + my ($self, $message, $level, $critical) = @_; + + # Assessment report are dumped to stdin so avoid printing debug info + return if (!$critical && $self->{type} eq 'SHOW_REPORT'); + + $level ||= 0; + + $message = '[' . strftime("%Y-%m-%d %H:%M:%S", localtime(time)) . '] ' . $message if ($self->{debug}); + if ($self->{debug} >= $level) { + if (defined $self->{fhlog}) { + $self->{fhlog}->print($message); + } else { + print $message; + } + } + if ($critical) { + if ($self->{debug} < $level) { + if (defined $self->{fhlog}) { + $self->{fhlog}->print($message); + } else { + print "$message\n"; + } + } + $self->{fhlog}->close() if (defined $self->{fhlog}); + $self->{dbh}->disconnect() if ($self->{dbh}); + $self->{dbhdest}->disconnect() if ($self->{dbhdest}); + die "Aborting export...\n"; + } +} + +=head2 logrep + +This function log report's information to STDOUT or to a logfile. + +=cut + +sub logrep +{ + my ($self, $message) = @_; + + if (defined $self->{fhlog}) { + $self->{fhlog}->print($message); + } else { + print $message; + } +} + + +=head2 _convert_type + +This function is used to rewrite Oracle TYPE DDL + +=cut + +sub _convert_type +{ + my ($self, $plsql, $owner, %pkg_type) = @_; + + my $unsupported = "-- Unsupported, please edit to match PostgreSQL syntax\n"; + my $content = ''; + my $type_name = ''; + + # Replace SUBTYPE declaration into DOMAIN declaration + if ($plsql =~ s/SUBTYPE\s+/CREATE DOMAIN /i) { + $plsql =~ s/\s+IS\s+/ AS /; + $plsql = Ora2Pg::PLSQL::replace_sql_type($plsql, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + return $plsql; + } + + $plsql =~ s/\s*INDEX\s+BY\s+([^\s;]+)//is; + if ($plsql =~ /TYPE\s+([^\s]+)\s+(IS|AS)\s*TABLE\s*OF\s+(.*)/is) { + $type_name = $1; + my $type_of = $3; + $type_name =~ s/"//g; + my $internal_name = $type_name; + if ($self->{export_schema} && !$self->{schema} && $owner) { + $type_name = "$owner.$type_name"; + } + $internal_name =~ s/^[^\.]+\.//; + $type_of =~ s/\s*NOT[\t\s]+NULL//is; + $type_of =~ s/\s*;\s*$//s; + $type_of =~ s/^\s+//s; + if ($type_of !~ /\s/s) { + $type_of = Ora2Pg::PLSQL::replace_sql_type($type_of, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + $self->{type_of_type}{'Nested Tables'}++; + $content = "CREATE TYPE \L$type_name\E AS (\L$internal_name\E $type_of\[\]);\n"; + } else { + $self->{type_of_type}{'Associative Arrays'}++; + $self->logit("WARNING: this kind of Nested Tables are not supported, skipping type $1\n", 1); + return "${unsupported}CREATE$self->{create_or_replace} $plsql"; + } + } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*REF\s+CURSOR/is) { + $self->logit("WARNING: TYPE REF CURSOR are not supported, skipping type $1\n", 1); + $plsql =~ s/\bREF\s+CURSOR/REFCURSOR/is; + $self->{type_of_type}{'Type Ref Cursor'}++; + return "${unsupported}CREATE$self->{create_or_replace} $plsql"; + } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*OBJECT\s*\((.*?)(TYPE BODY.*)/is) { + $self->{type_of_type}{'Type Boby'}++; + $self->logit("WARNING: TYPE BODY are not supported, skipping type $1\n", 1); + return "${unsupported}CREATE$self->{create_or_replace} $plsql"; + } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*(?:OBJECT|RECORD)\s*\((.*)\)([^\)]*)/is) { + $type_name = $1; + my $description = $3; + my $notfinal = $4; + $notfinal =~ s/\s+/ /gs; + if ($self->{export_schema} && !$self->{schema} && $owner) { + $type_name = "$owner.$type_name"; + } + if ($description =~ /\s*(MAP MEMBER|MEMBER|CONSTRUCTOR)\s+(FUNCTION|PROCEDURE).*/is) { + $self->{type_of_type}{'Type with member method'}++; + $self->logit("WARNING: TYPE with CONSTRUCTOR and MEMBER FUNCTION are not supported, skipping type $type_name\n", 1); + return "${unsupported}CREATE$self->{create_or_replace} $plsql"; + } + $description =~ s/^\s+//s; + my $declar = Ora2Pg::PLSQL::replace_sql_type($description, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + $type_name =~ s/"//g; + $type_name = $self->get_replaced_tbname($type_name); + if ($notfinal =~ /FINAL/is) { + $content = "-- Inherited types are not supported in PostgreSQL, replacing with inherited table\n"; + $content .= qq{CREATE TABLE $type_name ( +$declar +); +}; + $self->{type_of_type}{'Type inherited'}++; + } else { + $content = qq{ +CREATE TYPE $type_name AS ( +$declar +); +}; + $self->{type_of_type}{'Object type'}++; + } + } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+UNDER\s*([^\s]+)\s+\((.*)\)([^\)]*)/is) { + $type_name = $1; + my $type_inherit = $2; + my $description = $3; + if ($self->{export_schema} && !$self->{schema} && $owner) { + $type_name = "$owner.$type_name"; + } + if ($description =~ /\s*(MAP MEMBER|MEMBER|CONSTRUCTOR)\s+(FUNCTION|PROCEDURE).*/is) { + $self->logit("WARNING: TYPE with CONSTRUCTOR and MEMBER FUNCTION are not supported, skipping type $type_name\n", 1); + $self->{type_of_type}{'Type with member method'}++; + return "${unsupported}CREATE$self->{create_or_replace} $plsql"; + } + $description =~ s/^\s+//s; + my $declar = Ora2Pg::PLSQL::replace_sql_type($description, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + $type_name =~ s/"//g; + $type_name = $self->get_replaced_tbname($type_name); + $content = qq{ +CREATE TABLE $type_name ( +$declar +) INHERITS (\L$type_inherit\E); +}; + $self->{type_of_type}{'Subtype'}++; + } elsif ($plsql =~ /TYPE\s+([^\s]+)\s+(AS|IS)\s*(VARRAY|VARYING ARRAY)\s*\((\d+)\)\s*OF\s*(.*)/is) { + $type_name = $1; + my $size = $4; + my $tbname = $5; + $type_name =~ s/"//g; + $tbname =~ s/;//g; + my $internal_name = $type_name; + chomp($tbname); + if ($self->{export_schema} && !$self->{schema} && $owner) { + $type_name = "$owner.$type_name"; + } + $internal_name =~ s/^[^\.]+\.//; + my $declar = Ora2Pg::PLSQL::replace_sql_type($tbname, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + $declar =~ s/[\n\r]+//s; + $content = qq{ +CREATE TYPE \L$type_name\E AS ($internal_name $declar\[$size\]); +}; + $self->{type_of_type}{Varrays}++; + } else { + $self->{type_of_type}{Unknown}++; + $plsql =~ s/;$//s; + $content = "${unsupported}CREATE$self->{create_or_replace} $plsql;" + } + + if ($self->{force_owner}) { + $owner = $self->{force_owner} if ($self->{force_owner} ne "1"); + if ($owner) { + $content .= "ALTER TYPE " . $self->quote_object_name($type_name) + . " OWNER TO " . $self->quote_object_name($owner) . ";\n"; + } + } + + # Prefix type with their own package name + foreach my $t (keys %pkg_type) { + $content =~ s/(\s+)($t)\b/$1$pkg_type{$2}/igs; + } + + return $content; +} + +sub ask_for_data +{ + my ($self, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name, $is_subpart) = @_; + + # Build SQL query to retrieve data from this table + if (!$part_name) { + $self->logit("Looking how to retrieve data from $table...\n", 1); + } elsif ($is_subpart) { + $self->logit("Looking how to retrieve data from $table subpartition $part_name...\n", 1); + } else { + $self->logit("Looking how to retrieve data from $table partition $part_name...\n", 1); + } + my $query = $self->_howto_get_data($table, $nn, $tt, $stt, $part_name, $is_subpart); + + # Query with no column + if (!$query) { + $self->logit("WARNING: can not extract data from $table, no column found...\n", 0); + return 0; + } + + # Check for boolean rewritting + for (my $i = 0; $i <= $#{$nn}; $i++) { + my $colname = $nn->[$i]->[0]; + $colname =~ s/["`]//g; + my $typlen = $nn->[$i]->[5]; + $typlen ||= $nn->[$i]->[2]; + # Check if this column should be replaced by a boolean following table/column name + if (grep(/^$colname$/i, @{$self->{'replace_as_boolean'}{uc($table)}})) { + $tt->[$i] = 'boolean'; + # Check if this column should be replaced by a boolean following type/precision + } elsif (exists $self->{'replace_as_boolean'}{uc($nn->[$i]->[1])} && ($self->{'replace_as_boolean'}{uc($nn->[$i]->[1])}[0] == $typlen)) { + $tt->[$i] = 'boolean'; + } + } + + # check if destination column type must be changed + for (my $i = 0; $i <= $#{$nn}; $i++) { + my $colname = $nn->[$i]->[0]; + $colname =~ s/["`]//g; + $tt->[$i] = $self->{'modify_type'}{"\L$table\E"}{"\L$colname\E"} if (exists $self->{'modify_type'}{"\L$table\E"}{"\L$colname\E"}); + } + + # Look for user defined type + if (!$self->{is_mysql}) + { + for (my $idx = 0; $idx < scalar(@$stt); $idx++) + { + my $data_type = uc($stt->[$idx]) || ''; + $data_type =~ s/\(.*//; # remove any precision + # in case of user defined type try to gather the underlying base types + if (!exists $self->{data_type}{$data_type} && !exists $self->{user_type}{$data_type} + && $data_type !~ /SDO_GEOMETRY/i + && $data_type !~ /^(ST_|STGEOM_)/i #ArGis geometry types + ) { + %{ $self->{user_type}{$data_type} } = $self->custom_type_definition($data_type); + } + } + } + + if ( ($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"} ) { + $self->{ora_conn_count} = 0; + while ($self->{ora_conn_count} < $self->{oracle_copies}) { + spawn sub { + $self->logit("Creating new connection to database to extract data...\n", 1); + $self->_extract_data($query, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name, $self->{ora_conn_count}); + }; + $self->{ora_conn_count}++; + } + # Wait for oracle connection terminaison + while ($self->{ora_conn_count} > 0) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $self->{ora_conn_count}--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + if (defined $pipe) { + my $t_name = $part_name || $table; + my $t_time = time(); + $pipe->print("TABLE EXPORT ENDED: $t_name, end: $t_time, report all parts\n"); + } + } else { + my $total_record = $self->_extract_data($query, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name); + # Only useful for single process + return $total_record; + } + + return; +} + +sub custom_type_definition +{ + my ($self, $custom_type, $parent, $is_nested) = @_; + + my %user_type = (); + my $orig = $custom_type; + + my $data_type = uc($custom_type) || ''; + $data_type =~ s/\(.*//; # remove any precision + if (!exists $self->{data_type}{$data_type}) + { + if (!$is_nested) { + $self->logit("Data type $custom_type is not native, searching on custom types.\n", 1); + } else { + $self->logit("\tData type $custom_type nested from type $parent is not native, searching on custom types.\n", 1); + } + $custom_type = $self->_get_types($custom_type); + foreach my $tpe (sort {length($a->{name}) <=> length($b->{name}) } @{$custom_type}) + { + $self->logit("\tLooking inside custom type $tpe->{name} to extract values...\n", 1); + my %types_def = $self->_get_custom_types($tpe->{code}); + if ($#{$types_def{pg_types}} >= 0) + { + $self->logit("\tfound type description: $tpe->{name}(" . join(',', @{$types_def{pg_types}}) . ")\n", 1); + push(@{$user_type{pg_types}} , \@{$types_def{pg_types}}); + push(@{$user_type{src_types}}, \@{$types_def{src_types}}); + } + else + { + if ($tpe->{code} =~ /AS\s+VARRAY\s*(.*?)\s+OF\s+([^\s;]+);/is) { + return $self->custom_type_definition(uc($2), $orig, 1); + } + elsif ($tpe->{code} =~ /\s+([^\s]+)\s+AS\s+TABLE\s+OF\s+([^;]+);/is) + { + %types_def = $self->_get_custom_types("varname $2"); + push(@{$user_type{pg_types}} , \@{$types_def{pg_types}}); + push(@{$user_type{src_types}}, \@{$types_def{src_types}}); + } + else { + $self->logit("\tCan not found subtype for $tpe->{name} into code: $tpe->{code}\n", 1); + } + } + } + } + + return %user_type; +} + +sub _extract_data +{ + my ($self, $query, $table, $cmd_head, $cmd_foot, $s_out, $nn, $tt, $sprep, $stt, $part_name, $proc) = @_; + + $0 = "ora2pg - querying table $table"; + + # Overwrite the query if REPLACE_QUERY is defined for this table + if ($self->{replace_query}{"\L$table\E"}) { + $query = $self->{replace_query}{"\L$table\E"}; + if (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) { + my $colpk = $self->{defined_pk}{"\L$table\E"}; + if ($self->{preserve_case}) { + $colpk = '"' . $colpk . '"'; + } + my $cond = " ABS(MOD($colpk, $self->{oracle_copies})) = ?"; + if ($query !~ s/\bWHERE\s+/WHERE $cond AND /) { + if ($query !~ s/\b(ORDER\s+BY\s+.*)/WHERE $cond $1/) { + $query .= " WHERE $cond"; + } + } + } + } + + my %user_type = (); + my $rname = $part_name || $table; + my $dbh = 0; + my $sth = 0; + my @has_custom_type = (); + $self->{data_cols}{$table} = (); + + if ($self->{is_mysql}) { + my %col_info = Ora2Pg::MySQL::_column_info($self, $rname); + foreach my $col (keys %{$col_info{$rname}}) { + push(@{$self->{data_cols}{$table}}, $col); + } + } + + # Look for user defined type + if (!$self->{is_mysql}) { + for (my $idx = 0; $idx < scalar(@$stt); $idx++) { + my $data_type = uc($stt->[$idx]) || ''; + $data_type =~ s/\(.*//; # remove any precision + # in case of user defined type try to gather the underlying base types + if (!exists $self->{data_type}{$data_type} && exists $self->{user_type}{$stt->[$idx]}) { + push(@has_custom_type, $idx); + %{ $user_type{$idx} } = %{ $self->{user_type}{$stt->[$idx]} }; + } + } + } + + if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { + + $self->logit("DEBUG: cloning Oracle database connection.\n", 1); + $dbh = $self->{dbh}->clone(); + + # Force execution of initial command + $self->_ora_initial_command($dbh); + if (!$self->{is_mysql}) { + # Force numeric format into the cloned session + $self->_numeric_format($dbh); + # Force datetime format into the cloned session + $self->_datetime_format($dbh); + # Set the action name on Oracle side to see which table is exported + $dbh->do("CALL DBMS_APPLICATION_INFO.set_action('$table')") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } + + # Set row cache size + $dbh->{RowCacheSize} = int($self->{data_limit}/10); + if (exists $self->{local_data_limit}{$table}) { + $dbh->{RowCacheSize} = $self->{local_data_limit}{$table}; + } + + # prepare the query before execution + if (!$self->{is_mysql}) { + if ($self->{no_lob_locator}) { + $sth = $dbh->prepare($query,{ora_piece_lob => 1, ora_piece_size => $self->{longreadlen}, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } else { + $sth = $dbh->prepare($query,{'ora_auto_lob' => 0, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } + foreach (@{$sth->{NAME}}) { + push(@{$self->{data_cols}{$table}}, $_); + } + } else { + #$query .= " LIMIT ?, ?"; + $query =~ s/^SELECT\s+/SELECT \/\*\!40001 SQL_NO_CACHE \*\/ /s; + $sth = $dbh->prepare($query, { mysql_use_result => 1, mysql_use_row => 1 }) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } + + } else { + + # Set row cache size + $self->{dbh}->{RowCacheSize} = int($self->{data_limit}/10); + if (exists $self->{local_data_limit}{$table}) { + $self->{dbh}->{RowCacheSize} = $self->{local_data_limit}{$table}; + } + + # prepare the query before execution + if (!$self->{is_mysql}) + { + # Set the action name on Oracle side to see which table is exported + $self->{dbh}->do("CALL DBMS_APPLICATION_INFO.set_action('$table')") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + if ($self->{no_lob_locator}) { + $sth = $self->{dbh}->prepare($query,{ora_piece_lob => 1, ora_piece_size => $self->{longreadlen}, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}); + } else { + $sth = $self->{dbh}->prepare($query,{'ora_auto_lob' => 0, ora_exe_mode=>OCI_STMT_SCROLLABLE_READONLY, ora_check_sql => 1}); + } + + if ($self->{dbh}->errstr =~ /ORA-00942/) { + $self->logit("WARNING: table $table is not yet physically created and has no data.\n", 0, 0); + + # Only useful for single process + return 0; + } elsif ($self->{dbh}->errstr) { + $self->logit("FATAL: _extract_data() " . $self->{dbh}->errstr . "\n", 1, 1); + } + foreach (@{$sth->{NAME}}) { + push(@{$self->{data_cols}{$table}}, $_); + } + } else { + #$query .= " LIMIT ?, ?"; + $query =~ s/^SELECT\s+/SELECT \/\*\!40001 SQL_NO_CACHE \*\/ /s; + $sth = $self->{dbh}->prepare($query, { mysql_use_result => 1, mysql_use_row => 1 }) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + + } + + # Extract data now by chunk of DATA_LIMIT and send them to a dedicated job + $self->logit("Fetching all data from $rname tuples...\n", 1); + + my $start_time = time(); + my $total_record = 0; + my $total_row = $self->{tables}{$table}{table_info}{num_rows}; + + # Send current table in progress + if (defined $pipe) { + my $t_name = $part_name || $table; + if ($proc ne '') { + $pipe->print("TABLE EXPORT IN PROGESS: $t_name-part-$proc, start: $start_time, rows $total_row\n"); + } else { + $pipe->print("TABLE EXPORT IN PROGESS: $t_name, start: $start_time, rows $total_row\n"); + } + } + + my @params = (); + if (defined $proc) { + unshift(@params, $proc); + $self->logit("Parallelizing on core #$proc with query: $query\n", 1); + } + if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { + $sth->execute(@params) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } else { + $sth->execute(@params) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + + my $col_cond = $self->hs_cond($tt,$stt, $table); + + # Oracle allow direct retreiving of bchunk of data + if (!$self->{is_mysql}) { + + my $data_limit = $self->{data_limit}; + if (exists $self->{local_data_limit}{$table}) { + $data_limit = $self->{local_data_limit}{$table}; + } + my $has_blob = 0; + $has_blob = 1 if (grep(/LOB|XMLTYPE/, @$stt)); + + # With rows that not have custom type nor blob unless the user doesn't want to use lob locator + if (($#has_custom_type == -1) && (!$has_blob || $self->{no_lob_locator})) { + + while ( my $rows = $sth->fetchall_arrayref(undef,$data_limit)) { + if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { + if ($dbh->errstr) { + $self->logit("ERROR: " . $dbh->errstr . "\n", 0, 0); + last; + } + } elsif ( $self->{dbh}->errstr ) { + $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0); + last; + } + + $total_record += @$rows; + $self->{current_total_row} += @$rows; + $self->logit("DEBUG: number of rows $total_record extracted from table $table\n", 1); + + # Do we just want to test Oracle output speed + next if ($self->{oracle_speed} && !$self->{ora2pg_speed}); + + if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) { + while ($self->{child_count} >= $self->{jobs}) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $self->{child_count}--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + spawn sub { + $self->_dump_to_pg($proc, $rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); + }; + $self->{child_count}++; + } else { + $self->_dump_to_pg($proc, $rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); + } + } + + } else { + + my @rows = (); + while ( my @row = $sth->fetchrow_array()) + { + if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { + if ($dbh->errstr) { + $self->logit("ERROR: " . $dbh->errstr . "\n", 0, 0); + last; + } + } elsif ( $self->{dbh}->errstr ) { + $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0); + last; + } + + # Then foreach row use the returned lob locator to retrieve data + # and all column with a LOB data type, extract data by chunk + for (my $j = 0; $j <= $#$stt; $j++) + { + # Look for data based on custom type to replace the reference by the value + if ($row[$j] =~ /^(?!(?!)\x{100})ARRAY\(0x/ && $stt->[$j] !~ /SDO_GEOMETRY/i) + { + my $data_type = uc($stt->[$j]) || ''; + $data_type =~ s/\(.*//; # remove any precision + $row[$j] = $self->set_custom_type_value($data_type, $user_type{$j}, $row[$j], $tt->[$j], 0); + } + # Retrieve LOB data from locator + elsif (($stt->[$j] =~ /LOB|XMLTYPE/) && $row[$j]) + { + my $lob_content = ''; + my $offset = 1; # Offsets start at 1, not 0 + if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) + { + # Get chunk size + my $chunk_size = $self->{lob_chunk_size} || $dbh->ora_lob_chunk_size($row[$j]) || 8192; + while (1) + { + my $lobdata = $dbh->ora_lob_read($row[$j], $offset, $chunk_size ); + if ($dbh->errstr) { + $self->logit("ERROR: " . $dbh->errstr . "\n", 0, 0) if ($dbh->errstr !~ /ORA-22831/); + last; + } + last unless (defined $lobdata && length $lobdata); + $offset += $chunk_size; + $lob_content .= $lobdata; + } + } + else + { + # Get chunk size + my $chunk_size = $self->{lob_chunk_size} || $self->{dbh}->ora_lob_chunk_size($row[$j]) || 8192; + while (1) + { + my $lobdata = $self->{dbh}->ora_lob_read($row[$j], $offset, $chunk_size ); + if ($self->{dbh}->errstr) + { + $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0) if ($dbh->errstr !~ /ORA-22831/); + last; + } + last unless (defined $lobdata && length $lobdata); + $offset += $chunk_size; + $lob_content .= $lobdata; + } + } + if ($lob_content ne '') { + $row[$j] = $lob_content; + } else { + $row[$j] = undef; + } + + } + elsif (($stt->[$j] =~ /LOB/) && !$row[$j]) + { + # This might handle case where the LOB is NULL and might prevent error: + # DBD::Oracle::db::ora_lob_read: locator is not of type OCILobLocatorPtr + $row[$j] = undef; + } + } + $total_record++; + $self->{current_total_row}++; + + # Do we just want to test Oracle output speed + next if ($self->{oracle_speed} && !$self->{ora2pg_speed}); + + push(@rows, [ @row ] ); + + if ($#rows == $data_limit) + { + if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) { + while ($self->{child_count} >= $self->{jobs}) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $self->{child_count}--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + spawn sub { + $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); + }; + $self->{child_count}++; + } else { + $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); + } + @rows = (); + } + } + + # Do we just want to test Oracle output speed + next if ($self->{oracle_speed} && !$self->{ora2pg_speed}); + + # Flush last extracted data + if ( ($self->{jobs} > 1) || ($self->{oracle_copies} > 1) ) { + while ($self->{child_count} >= $self->{jobs}) { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $self->{child_count}--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + spawn sub { + $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); + }; + $self->{child_count}++; + } else { + $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); + } + @rows = (); + + } + + } else { + + my @rows = (); + my $num_row = 0; + while (my @row = $sth->fetchrow()) + { + push(@rows, \@row); + $num_row++; + if ($num_row == $self->{data_limit}) + { + $num_row = 0; + $total_record += @rows; + $self->{current_total_row} += @rows; + # Do we just want to test Oracle output speed + next if ($self->{oracle_speed} && !$self->{ora2pg_speed}); +# if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { +# my $max_jobs = $self->{jobs}; +# while ($self->{child_count} >= $max_jobs) { +# my $kid = waitpid(-1, WNOHANG); +# if ($kid > 0) { +# $self->{child_count}--; +# delete $RUNNING_PIDS{$kid}; +# } +# usleep(50000); +# } +# spawn sub { +# $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); +# }; +# $self->{child_count}++; +# } else { + $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); +# } + @rows = (); + } + } + + if (@rows && (!$self->{oracle_speed} || $self->{ora2pg_speed})) { + $total_record += @rows; + $self->{current_total_row} += @rows; +# if ( ($self->{parallel_tables} > 1) || (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) ) { +# my $max_jobs = $self->{jobs}; +# while ($self->{child_count} >= $max_jobs) { +# my $kid = waitpid(-1, WNOHANG); +# if ($kid > 0) { +# $self->{child_count}--; +# delete $RUNNING_PIDS{$kid}; +# } +# usleep(50000); +# } +# spawn sub { +# $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); +# }; +# $self->{child_count}++; +# } else { + $self->_dump_to_pg($proc, \@rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $start_time, $part_name, $total_record, %user_type); +# } + } + } + + $sth->finish(); + + # Close global data file in use when parallel table is used without output mutliprocess + $self->close_export_file($self->{cfhout}) if (defined $self->{cfhout}); + $self->{cfhout} = undef; + + if ( ($self->{jobs} <= 1) && ($self->{oracle_copies} <= 1) && ($self->{parallel_tables} <= 1)) + { + my $end_time = time(); + my $dt = $end_time - $self->{global_start_time}; + my $rps = int($self->{current_total_row} / ($dt||1)); + print STDERR "\n"; + print STDERR $self->progress_bar($self->{current_total_row}, $self->{global_rows}, 25, '=', 'total rows', "- ($dt sec., avg: $rps recs/sec).") . "\n"; + } + + # Wait for all child end + while ($self->{child_count} > 0) + { + my $kid = waitpid(-1, WNOHANG); + if ($kid > 0) { + $self->{child_count}--; + delete $RUNNING_PIDS{$kid}; + } + usleep(50000); + } + + if (defined $pipe) + { + my $t_name = $part_name || $table; + my $t_time = time(); + if ($proc ne '') { + $pipe->print("TABLE EXPORT ENDED: $t_name-part-$proc, end: $t_time, rows $total_record\n"); + } else { + $pipe->print("TABLE EXPORT ENDED: $t_name, end: $t_time, rows $total_record\n"); + } + } + + $dbh->disconnect() if ($dbh); + + # Only useful for single process + return $total_record; +} + +sub log_error_copy +{ + my ($self, $table, $s_out, $rows) = @_; + + my $outfile = ''; + if ($self->{output_dir} && !$noprefix) { + $outfile = $self->{output_dir} . '/'; + } + $outfile .= $table . '_error.log'; + + my $filehdl = new IO::File; + $filehdl->open(">>$outfile") or $self->logit("FATAL: Can't write to $outfile: $!\n", 0, 1); + $filehdl->print($s_out); + foreach my $row (@$rows) { + $filehdl->print(join("\t", @$row) . "\n"); + } + $filehdl->print("\\.\n"); + $self->close_export_file($filehdl); +} + +sub log_error_insert +{ + my ($self, $table, $sql_out) = @_; + + my $outfile = ''; + if ($self->{output_dir} && !$noprefix) { + $outfile = $self->{output_dir} . '/'; + } + $outfile .= $table . '_error.log'; + + my $filehdl = new IO::File; + $filehdl->open(">>$outfile") or $self->logit("FATAL: Can't write to $outfile: $!\n", 0, 1); + $filehdl->print("$sql_out\n"); + $self->close_export_file($filehdl); +} + + +sub _dump_to_pg +{ + my ($self, $procnum, $rows, $table, $cmd_head, $cmd_foot, $s_out, $tt, $sprep, $stt, $ora_start_time, $part_name, $glob_total_record, %user_type) = @_; + + my @tempfiles = (); + + if ($^O !~ /MSWin32|dos/i) { + push(@tempfiles, [ tempfile('tmp_ora2pgXXXXXX', SUFFIX => '', DIR => $TMP_DIR, UNLINK => 1 ) ]); + } + + # Oracle source table or partition + my $rname = $part_name || $table; + # Destination PostgreSQL table (direct import to partition is not allowed with native partitioning) + my $dname = $table; + $dname = $part_name if (!$self->{pg_supports_partition}); + + if ($self->{pg_dsn}) + { + $0 = "ora2pg - sending data from table $rname to table $dname"; + } else { + $0 = "ora2pg - writing to file data from table $rname to table $dname"; + } + + # Connect to PostgreSQL if direct import is enabled + my $dbhdest = undef; + if ($self->{pg_dsn} && !$self->{oracle_speed}) + { + $dbhdest = $self->_send_to_pgdb(); + $self->logit("Dumping data from table $rname into PostgreSQL table $dname...\n", 1); + $self->logit("Setting client_encoding to $self->{client_encoding}...\n", 1); + my $s = $dbhdest->do( "SET client_encoding TO '\U$self->{client_encoding}\E';") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + if (!$self->{synchronous_commit}) { + $self->logit("Disabling synchronous commit when writing to PostgreSQL...\n", 1); + $s = $dbhdest->do("SET synchronous_commit TO off") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } + } + + # Build header of the file + my $h_towrite = ''; + foreach my $cmd (@$cmd_head) + { + if ($self->{pg_dsn} && !$self->{oracle_speed}) { + my $s = $dbhdest->do("$cmd") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } else { + $h_towrite .= "$cmd\n"; + } + } + + # Build footer of the file + my $e_towrite = ''; + foreach my $cmd (@$cmd_foot) + { + if ($self->{pg_dsn} && !$self->{oracle_speed}) + { + my $s = $dbhdest->do("$cmd") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } else { + $e_towrite .= "$cmd\n"; + } + } + + # Preparing data for output + if ( !$sprep && ($#{$rows} >= 0) ) { + my $data_limit = $self->{data_limit}; + if (exists $self->{local_data_limit}{$table}) { + $data_limit = $self->{local_data_limit}{$table}; + } + my $len = @$rows; + $self->logit("DEBUG: Formatting bulk of $data_limit data (real: $len rows) for PostgreSQL.\n", 1); + $self->format_data($rows, $tt, $self->{type}, $stt, \%user_type, $table); + } + + # Add COPY header to the output + my $sql_out = $s_out; + + # Creating output + my $data_limit = $self->{data_limit}; + if (exists $self->{local_data_limit}{$table}) + { + $data_limit = $self->{local_data_limit}{$table}; + } + $self->logit("DEBUG: Creating output for $data_limit tuples\n", 1); + if ($self->{type} eq 'COPY') + { + if ($self->{pg_dsn}) { + $sql_out =~ s/;$//; + if (!$self->{oracle_speed}) { + $self->logit("DEBUG: Sending COPY bulk output directly to PostgreSQL backend\n", 1); + $dbhdest->do($sql_out) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + $sql_out = ''; + my $skip_end = 0; + foreach my $row (@$rows) { + unless($dbhdest->pg_putcopydata(join("\t", @$row) . "\n")) { + if ($self->{log_on_error}) { + $self->logit("ERROR (log error enabled): " . $dbhdest->errstr . "\n", 0, 0); + $self->log_error_copy($table, $s_out, $rows); + $skip_end = 1; + last; + } else { + $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } + } + } + unless ($dbhdest->pg_putcopyend()) { + if ($self->{log_on_error}) { + $self->logit("ERROR (log error enabled): " . $dbhdest->errstr . "\n", 0, 0); + $self->log_error_copy($table, $s_out, $rows) if (!$skip_end); + } else { + $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } + } + } else { + foreach my $row (@$rows) { + # do nothing, just add loop time nothing must be sent to PG + } + } + } else { + # then add data to the output + map { $sql_out .= join("\t", @$_) . "\n"; } @$rows; + $sql_out .= "\\.\n"; + } + } + elsif (!$sprep) + { + $sql_out = ''; + foreach my $row (@$rows) { + $sql_out .= $s_out; + $sql_out .= join(',', @$row) . ");\n"; + } + } + + # Insert data if we are in online processing mode + if ($self->{pg_dsn}) + { + if ($self->{type} ne 'COPY') + { + if (!$sprep && !$self->{oracle_speed}) { + $self->logit("DEBUG: Sending INSERT output directly to PostgreSQL backend\n", 1); + unless($dbhdest->do("BEGIN;\n" . $sql_out . "COMMIT;\n")) { + if ($self->{log_on_error}) { + $self->logit("WARNING (log error enabled): " . $dbhdest->errstr . "\n", 0, 0); + $self->log_error_insert($table, "BEGIN;\n" . $sql_out . "COMMIT;\n"); + } else { + $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } + } + } else { + my $ps = undef; + if (!$self->{oracle_speed}) { + $ps = $dbhdest->prepare($sprep) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } + my @date_cols = (); + my @bool_cols = (); + for (my $i = 0; $i <= $#{$tt}; $i++) + { + if ($tt->[$i] eq 'bytea') { + if (!$self->{oracle_speed}) { + $ps->bind_param($i+1, undef, { pg_type => DBD::Pg::PG_BYTEA }); + } + } elsif ($tt->[$i] eq 'boolean') { + push(@bool_cols, $i); + } elsif ($tt->[$i] =~ /(date|time)/i) { + push(@date_cols, $i); + } + } + $self->logit("DEBUG: Sending INSERT bulk output directly to PostgreSQL backend\n", 1); + my $col_cond = $self->hs_cond($tt, $stt, $table); + foreach my $row (@$rows) + { + # Even with prepared statement we need to replace zero date + foreach my $j (@date_cols) { + if ($row->[$j] =~ /^0000-00-00/) { + if (!$self->{replace_zero_date}) { + $row->[$j] = undef; + } else { + $row->[$j] = $self->{replace_zero_date}; + } + } + } + # Format user defined type and geometry data + $self->format_data_row($row,$tt,'INSERT', $stt, \%user_type, $table, $col_cond, 1); + # Replace boolean 't' and 'f' by 0 and 1 for bind parameters. + foreach my $j (@bool_cols) { + ($row->[$j] eq "'f'") ? $row->[$j] = 0 : $row->[$j] = 1; + } + # Apply bind parmeters + if (!$self->{oracle_speed}) { + unless ($ps->execute(@$row) ) { + if ($self->{log_on_error}) { + $self->logit("ERROR (log error enabled): " . $ps->errstr . "\n", 0, 0); + $s_out =~ s/\([,\?]+\)/\(/; + $self->format_data_row($row,$tt,'INSERT', $stt, \%user_type, $table, $col_cond); + $self->log_error_insert($table, $s_out . join(',', @$row) . ");\n"); + } else { + $self->logit("FATAL: " . $ps->errstr . "\n", 0, 1); + } + } + } + } + if (!$self->{oracle_speed}) { + $ps->finish(); + } + } + } + } + else + { + if ($part_name && $self->{prefix_partition}) { + $part_name = $table . '_' . $part_name; + } + $sql_out = $h_towrite . $sql_out . $e_towrite; + if (!$self->{oracle_speed}) { + $self->data_dump($sql_out, $table, $part_name); + } + } + + my $total_row = $self->{tables}{$table}{table_info}{num_rows}; + my $tt_record = @$rows; + $dbhdest->disconnect() if ($dbhdest); + + my $end_time = time(); + $ora_start_time = $end_time if (!$ora_start_time); + my $dt = $end_time - $ora_start_time; + my $rps = int($glob_total_record / ($dt||1)); + my $t_name = $part_name || $table; + if (!$self->{quiet} && !$self->{debug}) + { + # Send current table in progress + if (defined $pipe) + { + if ($procnum ne '') + { + $pipe->print("CHUNK $$ DUMPED: $t_name-part-$procnum, time: $end_time, rows $tt_record\n"); + } + else + { + $pipe->print("CHUNK $$ DUMPED: $t_name, time: $end_time, rows $tt_record\n"); + } + } + else + { + print STDERR $self->progress_bar($glob_total_record, $total_row, 25, '=', 'rows', "Table $t_name ($rps recs/sec)"), "\r"; + } + } + elsif ($self->{debug}) + { + $self->logit("Extracted records from table $t_name: total_records = $glob_total_record (avg: $rps recs/sec)\n", 1); + } + + if ($^O !~ /MSWin32|dos/i) + { + if (defined $tempfiles[0]->[0]) + { + close($tempfiles[0]->[0]); + } + unlink($tempfiles[0]->[1]) if (-e $tempfiles[0]->[1]); + } +} + +sub _pload_to_pg +{ + my ($self, $idx, $query, @settings) = @_; + + if (!$self->{pg_dsn}) + { + $self->logit("FATAL: No connection to PostgreSQL database set, aborting...\n", 0, 1); + } + + my @tempfiles = (); + + if ($^O !~ /MSWin32|dos/i) + { + push(@tempfiles, [ tempfile('tmp_ora2pgXXXXXX', SUFFIX => '', DIR => $TMP_DIR, UNLINK => 1 ) ]); + } + + # Open a connection to the postgreSQL database + $0 = "ora2pg - sending query to PostgreSQL database"; + + # Connect to PostgreSQL if direct import is enabled + my $dbhdest = $self->_send_to_pgdb(); + $self->logit("Loading query #$idx: $query\n", 1); + if ($#settings == -1) + { + $self->logit("Applying settings from configuration\n", 1); + # Apply setting from configuration + $dbhdest->do( "SET client_encoding TO '\U$self->{client_encoding}\E';") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + my $search_path = $self->set_search_path(); + if ($search_path) { + $dbhdest->do($search_path) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } + } + else + { + $self->logit("Applying settings from input file\n", 1); + # Apply setting from source file + foreach my $set (@settings) { + $dbhdest->do($set) or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + } + } + # Execute query + $dbhdest->do("$query") or $self->logit("FATAL: " . $dbhdest->errstr . "\n", 0, 1); + $dbhdest->disconnect() if ($dbhdest); + + if ($^O !~ /MSWin32|dos/i) + { + if (defined $tempfiles[0]->[0]) + { + close($tempfiles[0]->[0]); + } + unlink($tempfiles[0]->[1]) if (-e $tempfiles[0]->[1]); + } +} + + +# Global array, to store the converted values +my @bytea_array; +sub build_escape_bytea +{ + foreach my $tmp (0..255) + { + my $out; + if ($tmp >= 32 and $tmp <= 126) { + if ($tmp == 92) { + $out = '\\\\134'; + } elsif ($tmp == 39) { + $out = '\\\\047'; + } else { + $out = chr($tmp); + } + } else { + $out = sprintf('\\\\%03o',$tmp); + } + $bytea_array[$tmp] = $out; + } +} + +=head2 escape_bytea + +This function return an escaped bytea entry for Pg. + +=cut + + +sub escape_bytea +{ + my $data = shift; + + # In this function, we use the array built by build_escape_bytea + my @array= unpack("C*", $data); + foreach my $elt (@array) { + $elt = $bytea_array[$elt]; + } + return join('', @array); +} + +=head2 _show_infos + +This function display a list of schema, table or column only to stdout. + +=cut + +sub _show_infos +{ + my ($self, $type) = @_; + + if ($type eq 'SHOW_ENCODING') + { + if ($self->{is_mysql}) + { + $self->logit("Current encoding settings that will be used by Ora2Pg:\n", 0); + $self->logit("\tMySQL database and client encoding: $self->{nls_lang}\n", 0); + $self->logit("\tMySQL collation encoding: $self->{nls_nchar}\n", 0); + $self->logit("\tPostgreSQL CLIENT_ENCODING $self->{client_encoding}\n", 0); + $self->logit("\tPerl output encoding '$self->{binmode}'\n", 0); + my ($my_encoding, $my_client, $pg_encoding, $my_timestamp_format, $my_date_format) = &Ora2Pg::MySQL::_get_encoding($self, $self->{dbh}); + $self->logit("Showing current MySQL encoding and possible PostgreSQL client encoding:\n", 0); + $self->logit("\tMySQL database and client encoding: $my_encoding\n", 0); + $self->logit("\tMySQL collation encoding: $my_client\n", 0); + $self->logit("\tPostgreSQL CLIENT_ENCODING: $pg_encoding\n", 0); + $self->logit("MySQL SQL mode: $self->{mysql_mode}\n", 0); + } else { + $self->logit("Current encoding settings that will be used by Ora2Pg:\n", 0); + $self->logit("\tOracle NLS_LANG $self->{nls_lang}\n", 0); + $self->logit("\tOracle NLS_NCHAR $self->{nls_nchar}\n", 0); + if ($self->{enable_microsecond}) { + my $dim = 6; + $dim = '' if ($self->{db_version} =~ /Release  [89]/); + $self->logit("\tOracle NLS_TIMESTAMP_FORMAT YYYY-MM-DD HH24:MI:SS.FF$dim\n", 0); + } else { + $self->logit("\tOracle NLS_TIMESTAMP_FORMAT YYYY-MM-DD HH24:MI:SS\n", 0); + } + $self->logit("\tOracle NLS_DATE_FORMAT YYYY-MM-DD HH24:MI:SS\n", 0); + $self->logit("\tPostgreSQL CLIENT_ENCODING $self->{client_encoding}\n", 0); + $self->logit("\tPerl output encoding '$self->{binmode}'\n", 0); + + my ($ora_encoding, $ora_charset, $pg_encoding, $nls_timestamp_format, $nls_date_format) = $self->_get_encoding($self->{dbh}); + $self->logit("Showing current Oracle encoding and possible PostgreSQL client encoding:\n", 0); + $self->logit("\tOracle NLS_LANG $ora_encoding\n", 0); + $self->logit("\tOracle NLS_NCHAR $ora_charset\n", 0); + $self->logit("\tOracle NLS_TIMESTAMP_FORMAT $nls_timestamp_format\n", 0); + $self->logit("\tOracle NLS_DATE_FORMAT $nls_date_format\n", 0); + $self->logit("\tPostgreSQL CLIENT_ENCODING $pg_encoding\n", 0); + } + } + elsif ($type eq 'SHOW_VERSION') + { + $self->logit("Showing Database Version...\n", 1); + $self->logit("$self->{db_version}\n", 0); + } + elsif ($type eq 'SHOW_REPORT') + { + print STDERR "Reporting Oracle Content...\n" if ($self->{debug}); + my $uncovered_score = 'Ora2Pg::PLSQL::UNCOVERED_SCORE'; + if ($self->{is_mysql}) { + $uncovered_score = 'Ora2Pg::PLSQL::UNCOVERED_MYSQL_SCORE'; + } + # Get Oracle database version and size + print STDERR "Looking at Oracle server version...\n" if ($self->{debug}); + my $ver = $self->_get_version(); + print STDERR "Looking at Oracle database size...\n" if ($self->{debug}); + my $size = $self->_get_database_size(); + # Get the list of all database objects + print STDERR "Looking at Oracle defined objects...\n" if ($self->{debug}); + my %objects = $self->_get_objects(); + + # Extract all tables informations + my %all_indexes = (); + $self->{skip_fkeys} = $self->{skip_indices} = $self->{skip_indexes} = $self->{skip_checks} = 0; + $self->{view_as_table} = (); + print STDERR "Looking at table definition...\n" if ($self->{debug}); + $self->_tables(1); + my $total_index = 0; + my $total_table_objects = 0; + my $total_index_objects = 0; + foreach my $table (sort keys %{$self->{tables}}) + { + $total_table_objects++; + push(@exported_indexes, $self->_exportable_indexes($table, %{$self->{tables}{$table}{indexes}})); + $total_index_objects += scalar keys %{$self->{tables}{$table}{indexes}}; + foreach my $idx (sort keys %{$self->{tables}{$table}{idx_type}}) + { + next if (!grep(/^$idx$/i, @exported_indexes)); + my $typ = $self->{tables}{$table}{idx_type}{$idx}{type}; + push(@{$all_indexes{$typ}}, $idx); + $total_index++; + } + } + # Convert Oracle user defined type to PostgreSQL + if (!$self->{is_mysql}) { + $self->_types(); + foreach my $tpe (sort { $a->{pos} <=> $b->{pos} } @{$self->{types}}) { + # We dont want the result but only the array @{$self->{types}} + # define in the _convert_type() function + $self->_convert_type($tpe->{code}, $tpe->{owner}); + } + } + print STDERR "Looking at views definition...\n" if ($self->{debug}); + my %view_infos = (); + %view_infos = $self->_get_views() if ($self->{estimate_cost}); + + # Get definition of Database Link + print STDERR "Looking at database links...\n" if ($self->{debug}); + my %dblink = $self->_get_dblink(); + $objects{'DATABASE LINK'} = scalar keys %dblink; + print STDERR "\tFound $objects{'DATABASE LINK'} DATABASE LINK.\n" if ($self->{debug}); + # Get Jobs + print STDERR "Looking at jobs...\n" if ($self->{debug}); + my %jobs = $self->_get_job(); + $objects{'JOB'} = scalar keys %jobs; + print STDERR "\tFound $objects{'JOB'} JOB.\n" if ($self->{debug}); + # Get synonym information + print STDERR "Looking at synonyms...\n" if ($self->{debug}); + my %synonyms = $self->_synonyms(); + $objects{'SYNONYM'} = scalar keys %synonyms; + print STDERR "\tFound $objects{'SYNONYM'} SYNONYM.\n" if ($self->{debug}); + # Get all global temporary tables + print STDERR "Looking at global temporary table...\n" if ($self->{debug}); + my %global_tables = $self->_global_temp_table_info(); + $objects{'GLOBAL TEMPORARY TABLE'} = scalar keys %global_tables; + print STDERR "\tFound $objects{'GLOBAL TEMPORARY TABLE'} GLOBAL TEMPORARY TABLE.\n" if ($self->{debug}); + # Look for encrypted columns and identity columns + my %encrypted_column = (); + if ($self->{db_version} !~ /Release [89]/) { + print STDERR "Looking at encrypted columns...\n" if ($self->{debug}); + %encrypted_column = $self->_encrypted_columns('',$self->{schema}); + print STDERR "\tFound ", scalar keys %encrypted_column, " encrypted column.\n" if ($self->{debug}); + print STDERR "Looking at identity columns...\n" if ($self->{debug}); + # Identity column are collected in call to sub _tables() above + print STDERR "\tFound ", scalar keys %{$self->{identity_info}}, " identity column.\n" if ($self->{debug}); + } + + # Look at all database objects to compute report + my %report_info = (); + $report_info{'Version'} = $ver || 'Unknown'; + $report_info{'Schema'} = $self->{schema} || ''; + $report_info{'Size'} = $size || 'Unknown'; + my $idx = 0; + my $num_total_obj = scalar keys %objects; + foreach my $typ (sort keys %objects) + { + $idx++; + next if ($typ eq 'EVALUATION CONTEXT'); # Do not care about rule evaluation context + next if ($self->{is_mysql} && $typ eq 'SYNONYM'); + next if ($typ eq 'PACKAGE'); # Package are scanned with PACKAGE BODY not PACKAGE objects + print STDERR "Building report for object $typ...\n" if ($self->{debug}); + if (!$self->{quiet} && !$self->{debug}) { + print STDERR $self->progress_bar($idx, $num_total_obj, 25, '=', 'objects types', "inspecting object $typ" ), "\r"; + } + $report_info{'Objects'}{$typ}{'number'} = 0; + $report_info{'Objects'}{$typ}{'invalid'} = 0; + if (!grep(/^$typ$/, 'DATABASE LINK', 'JOB', 'TABLE', 'INDEX', + 'SYNONYM','GLOBAL TEMPORARY TABLE')) + { + for (my $i = 0; $i <= $#{$objects{$typ}}; $i++) + { + $report_info{'Objects'}{$typ}{'number'}++; + $report_info{'Objects'}{$typ}{'invalid'}++ if ($objects{$typ}[$i]->{invalid}); + } + } + elsif ($typ eq 'TABLE') + { + $report_info{'Objects'}{$typ}{'number'} = $total_table_objects; + } + elsif ($typ eq 'INDEX') + { + $report_info{'Objects'}{$typ}{'number'} = $total_index_objects; + } + else + { + $report_info{'Objects'}{$typ}{'number'} = $objects{$typ}; + } + + $report_info{'total_object_invalid'} += $report_info{'Objects'}{$typ}{'invalid'}; + $report_info{'total_object_number'} += $report_info{'Objects'}{$typ}{'number'}; + + if ($report_info{'Objects'}{$typ}{'number'} > 0) + { + $report_info{'Objects'}{$typ}{'real_number'} = ($report_info{'Objects'}{$typ}{'number'} - $report_info{'Objects'}{$typ}{'invalid'}); + $report_info{'Objects'}{$typ}{'real_number'} = $report_info{'Objects'}{$typ}{'number'} if ($self->{export_invalid}); + } + + if ($self->{estimate_cost}) + { + $report_info{'Objects'}{$typ}{'cost_value'} = ($report_info{'Objects'}{$typ}{'real_number'}*$Ora2Pg::PLSQL::OBJECT_SCORE{$typ}); + # Minimal unit is 1 + $report_info{'Objects'}{$typ}{'cost_value'} = 1 if ($report_info{'Objects'}{$typ}{'cost_value'} =~ /^0\./); + # For some object's type do not set migration unit upper than 2 days. + if (grep(/^$typ$/, 'TABLE PARTITION', 'GLOBAL TEMPORARY TABLE', 'TRIGGER', 'VIEW')) + { + $report_info{'Objects'}{$typ}{'cost_value'} = 168 if ($report_info{'Objects'}{$typ}{'cost_value'} > 168); + if (grep(/^$typ$/, 'TRIGGER', 'VIEW') && $report_info{'Objects'}{$typ}{'real_number'} > 500) { + $report_info{'Objects'}{$typ}{'cost_value'} += 84 * int(($report_info{'Objects'}{$typ}{'real_number'} - 500) / 500); + } + } + elsif (grep(/^$typ$/, 'TABLE', 'INDEX', 'SYNONYM')) + { + $report_info{'Objects'}{$typ}{'cost_value'} = 84 if ($report_info{'Objects'}{$typ}{'cost_value'} > 84); + } + } + + if ($typ eq 'INDEX') + { + my $bitmap = 0; + foreach my $t (sort keys %INDEX_TYPE) + { + my $len = ($#{$all_indexes{$t}}+1); + $report_info{'Objects'}{$typ}{'detail'} .= "\L$len $INDEX_TYPE{$t} index(es)\E\n" if ($len); + if ($self->{estimate_cost} && $len && + ( ($t =~ /FUNCTION.*NORMAL/) || ($t eq 'FUNCTION-BASED BITMAP') ) ) + { + $report_info{'Objects'}{$typ}{'cost_value'} += ($len * $Ora2Pg::PLSQL::OBJECT_SCORE{'FUNCTION-BASED-INDEX'}); + } + if ($self->{estimate_cost} && $len && ($t =~ /REV/)) { + $report_info{'Objects'}{$typ}{'cost_value'} += ($len * $Ora2Pg::PLSQL::OBJECT_SCORE{'REV-INDEX'}); + } + } + $report_info{'Objects'}{$typ}{'cost_value'} += ($Ora2Pg::PLSQL::OBJECT_SCORE{$typ}*$total_index) if ($self->{estimate_cost}); + $report_info{'Objects'}{$typ}{'comment'} = "$total_index index(es) are concerned by the export, others are automatically generated and will do so on PostgreSQL."; + my $hash_index = ''; + if ($self->{pg_version} < 10) + { + $hash_index = ' and hash index(es) will be exported as b-tree index(es) if any'; + } + if (!$self->{is_mysql}) { + my $bitmap = 'Bitmap'; + if ($self->{bitmap_as_gin}) { + $bitmap = 'Bitmap will be exported as btree_gin index(es)'; + } + $report_info{'Objects'}{$typ}{'comment'} .= " $bitmap$hash_index. Domain index are exported as b-tree but commented to be edited to mainly use FTS. Cluster, bitmap join and IOT indexes will not be exported at all. Reverse indexes are not exported too, you may use a trigram-based index (see pg_trgm) or a reverse() function based index and search. Use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns."; + } else { + $report_info{'Objects'}{$typ}{'comment'} .= "$hash_index. Use 'varchar_pattern_ops', 'text_pattern_ops' or 'bpchar_pattern_ops' operators in your indexes to improve search with the LIKE operator respectively into varchar, text or char columns. Fulltext search indexes will be replaced by using a dedicated tsvector column, Ora2Pg will set the DDL to create the column, function and trigger together with the index."; + } + } + elsif ($typ eq 'MATERIALIZED VIEW') + { + $report_info{'Objects'}{$typ}{'comment'}= "All materialized view will be exported as snapshot materialized views, they are only updated when fully refreshed."; + my %mview_infos = $self->_get_materialized_views(); + my $oncommit = 0; + foreach my $mview (sort keys %mview_infos) { + if ($mview_infos{$mview}{refresh_mode} eq 'COMMIT') { + $oncommit++; + $report_info{'Objects'}{$typ}{'detail'} .= "$mview, "; + } + } + if ($oncommit) { + $report_info{'Objects'}{$typ}{'detail'} =~ s/, $//; + $report_info{'Objects'}{$typ}{'detail'} = "$oncommit materialized views are refreshed on commit ($report_info{'Objects'}{$typ}{'detail'}), this is not supported by PostgreSQL, you will need to use triggers to have the same behavior or use a simple view."; + } + + + } + elsif ($typ eq 'TABLE') + { + my $exttb = scalar keys %{$self->{external_table}}; + if ($exttb) { + if (!$self->{external_to_fdw}) { + $report_info{'Objects'}{$typ}{'comment'} = "$exttb external table(s) will be exported as standard table. See EXTERNAL_TO_FDW configuration directive to export as file_fdw foreign tables or use COPY in your code if you just want to load data from external files."; + } else { + $report_info{'Objects'}{$typ}{'comment'} = "$exttb external table(s) will be exported as file_fdw foreign table. See EXTERNAL_TO_FDW configuration directive to export as standard table or use COPY in your code if you just want to load data from external files."; + } + } + + my %table_detail = (); + my $virt_column = 0; + my @done = (); + my $id = 0; + my $total_check = 0; + my $total_row_num = 0; + # Set the table information for each class found + foreach my $t (sort keys %{$self->{tables}}) + { + # Set the total number of rows + $total_row_num += $self->{tables}{$t}{table_info}{num_rows}; + + # Look at reserved words if tablename is found + my $r = $self->is_reserved_words($t); + if (($r > 0) && ($r != 3)) { + $table_detail{'reserved words in table name'}++; + $report_info{'Objects'}{$typ}{'cost_value'} += 12; # one hour to solve reserved keyword might be enough + } + # Get fields informations + foreach my $k (sort {$self->{tables}{$t}{column_info}{$a}[11] <=> $self->{tables}{$t}{column_info}{$a}[11]} keys %{$self->{tables}{$t}{column_info}}) + { + $r = $self->is_reserved_words($self->{tables}{$t}{column_info}{$k}[0]); + if (($r > 0) && ($r != 3)) { + $table_detail{'reserved words in column name'}++; + $report_info{'Objects'}{$typ}{'cost_value'} += 12; # one hour to solve reserved keyword might be enough + } elsif ($r == 3) { + $table_detail{'system columns in column name'}++; + $report_info{'Objects'}{$typ}{'cost_value'} += 12; # one hour to solve reserved keyword might be enough + } + $self->{tables}{$t}{column_info}{$k}[1] =~ s/TIMESTAMP\(\d+\)/TIMESTAMP/i; + if (!$self->{is_mysql}) { + if (!exists $self->{data_type}{uc($self->{tables}{$t}{column_info}{$k}[1])}) { + $table_detail{'unknown types'}++; + } + } else { + if (!exists $Ora2Pg::MySQL::MYSQL_TYPE{uc($self->{tables}{$t}{column_info}{$k}[1])}) { + $table_detail{'unknown types'}++; + } + } + if ( (uc($self->{tables}{$t}{column_info}{$k}[1]) eq 'NUMBER') && ($self->{tables}{$t}{column_info}{$k}[2] eq '') ) { + $table_detail{'numbers with no precision'}++; + } + if ( $self->{data_type}{uc($self->{tables}{$t}{column_info}{$k}[1])} eq 'bytea' ) { + $table_detail{'binary columns'}++; + } + } + # Get check constraints information related to this table + my $constraints = $self->_count_check_constraint($self->{tables}{$t}{check_constraint}); + $total_check += $constraints; + if ($self->{estimate_cost} && $constraints >= 0) { + $report_info{'Objects'}{$typ}{'cost_value'} += $constraints * $Ora2Pg::PLSQL::OBJECT_SCORE{'CHECK'}; + } + } + $report_info{'Objects'}{$typ}{'comment'} .= " $total_check check constraint(s)." if ($total_check); + foreach my $d (sort keys %table_detail) { + $report_info{'Objects'}{$typ}{'comment'} .= "\L$table_detail{$d} $d\E.\n"; + } + $report_info{'Objects'}{$typ}{'detail'} .= "Total number of rows: $total_row_num\n"; + $report_info{'Objects'}{$typ}{'detail'} .= "Top $self->{top_max} of tables sorted by number of rows:\n"; + my $j = 1; + foreach my $t (sort {$self->{tables}{$b}{table_info}{num_rows} <=> $self->{tables}{$a}{table_info}{num_rows}} keys %{$self->{tables}}) { + $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E has $self->{tables}{$t}{table_info}{num_rows} rows\n"; + $j++; + last if ($j > $self->{top_max}); + } + my %largest_table = (); + %largest_table = $self->_get_largest_tables() if ($self->{is_mysql}); + if ((scalar keys %largest_table > 0) || !$self->{is_mysql}) { + $i = 1; + if (!$self->{is_mysql}) { + $report_info{'Objects'}{$typ}{'detail'} .= "Top $self->{top_max} of largest tables:\n"; + foreach my $t (sort { $largest_table{$b} <=> $largest_table{$a} } keys %largest_table) { + $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E: $largest_table{$t} MB (" . $self->{tables}{$t}{table_info}{num_rows} . " rows)\n"; + $i++; + last if ($i > $self->{top_max}); + } + } else { + $report_info{'Objects'}{$typ}{'detail'} .= "Top $self->{top_max} of largest tables:\n"; + foreach my $t (sort {$self->{tables}{$b}{table_info}{size} <=> $self->{tables}{$a}{table_info}{size}} keys %{$self->{tables}}) { + $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E: $self->{tables}{$t}{table_info}{size} MB (" . $self->{tables}{$t}{table_info}{num_rows} . " rows)\n"; + $i++; + last if ($i > $self->{top_max}); + } + } + } + $comment = "Nothing particular." if (!$comment); + $report_info{'Objects'}{$typ}{'cost_value'} =~ s/(\.\d).*$/$1/; + if (scalar keys %encrypted_column > 0) { + $report_info{'Objects'}{$typ}{'comment'} .= "\n" . (scalar keys %encrypted_column) . " encrypted column(s).\n"; + foreach my $k (sort keys %encrypted_column) { + $report_info{'Objects'}{$typ}{'comment'} .= "\L$k\E ($encrypted_column{$k})\n"; + } + $report_info{'Objects'}{$typ}{'comment'} .= ". You must use the pg_crypto extension to use encryption.\n"; + if ($self->{estimate_cost}) { + $report_info{'Objects'}{$typ}{'cost_value'} += (scalar keys %encrypted_column) * $Ora2Pg::PLSQL::OBJECT_SCORE{'ENCRYPTED COLUMN'}; + } + } + if (scalar keys %{$self->{identity_info}} > 0) { + $report_info{'Objects'}{$typ}{'comment'} .= "\n" . (scalar keys %{$self->{identity_info}}) . " identity column(s).\n"; + $report_info{'Objects'}{$typ}{'comment'} .= " Identity columns are fully supported since PG10.\n"; + } + } + elsif ($typ eq 'TYPE') + { + my $total_type = $report_info{'Objects'}{'TYPE'}{'number'}; + foreach my $t (sort keys %{$self->{type_of_type}}) + { + $total_type-- if (grep(/^$t$/, 'Associative Arrays','Type Boby','Type with member method', 'Type Ref Cursor')); + $report_info{'Objects'}{$typ}{'detail'} .= "\L$self->{type_of_type}{$t} $t\E\n" if ($self->{type_of_type}{$t}); + } + $report_info{'Objects'}{$typ}{'cost_value'} = ($Ora2Pg::PLSQL::OBJECT_SCORE{$typ}*$total_type) if ($self->{estimate_cost}); + $report_info{'Objects'}{$typ}{'comment'} = "$total_type type(s) are concerned by the export, others are not supported. Note that Type inherited and Subtype are converted as table, type inheritance is not supported."; + } + elsif ($typ eq 'TYPE BODY') + { + $report_info{'Objects'}{$typ}{'comment'} = "Export of type with member method are not supported, they will not be exported."; + } + elsif ($typ eq 'TRIGGER') + { + my $triggers = $self->_get_triggers(); + my $total_size = 0; + foreach my $trig (@{$triggers}) { + $total_size += length($trig->[4]); + if ($self->{estimate_cost}) { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $trig->[4]); + $report_info{'Objects'}{$typ}{'cost_value'} += $cost; + $report_info{'Objects'}{$typ}{'detail'} .= "\L$trig->[0]: $cost\E\n"; + $report_info{full_trigger_details}{"\L$trig->[0]\E"}{count} = $cost; + foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { + next if (!$cost_detail{$d}); + $report_info{full_trigger_details}{"\L$trig->[0]\E"}{info} .= "\t$d => $cost_detail{$d}"; + $report_info{full_trigger_details}{"\L$trig->[0]\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); + $report_info{full_trigger_details}{"\L$trig->[0]\E"}{info} .= "\n"; + push(@{$report_info{full_trigger_details}{"\L$trig->[0]\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); + } + } + } + $report_info{'Objects'}{$typ}{'comment'} = "Total size of trigger code: $total_size bytes."; + } + elsif ($typ eq 'SEQUENCE') + { + $report_info{'Objects'}{$typ}{'comment'} = "Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name')."; + } + elsif ($typ eq 'FUNCTION') + { + my $functions = $self->_get_functions(); + my $total_size = 0; + foreach my $fct (keys %{$functions}) { + $total_size += length($functions->{$fct}{text}); + if ($self->{estimate_cost}) { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $functions->{$fct}{text}); + $report_info{'Objects'}{$typ}{'cost_value'} += $cost; + $report_info{'Objects'}{$typ}{'detail'} .= "\L$fct: $cost\E\n"; + $report_info{full_function_details}{"\L$fct\E"}{count} = $cost; + foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { + next if (!$cost_detail{$d}); + $report_info{full_function_details}{"\L$fct\E"}{info} .= "\t$d => $cost_detail{$d}"; + $report_info{full_function_details}{"\L$fct\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); + $report_info{full_function_details}{"\L$fct\E"}{info} .= "\n"; + push(@{$report_info{full_function_details}{"\L$fct\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); + } + } + } + $report_info{'Objects'}{$typ}{'comment'} = "Total size of function code: $total_size bytes."; + } + elsif ($typ eq 'PROCEDURE') + { + my $procedures = $self->_get_procedures(); + my $total_size = 0; + foreach my $proc (keys %{$procedures}) + { + $total_size += length($procedures->{$proc}{text}); + if ($self->{estimate_cost}) { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $procedures->{$proc}{text}); + $report_info{'Objects'}{$typ}{'cost_value'} += $cost; + $report_info{'Objects'}{$typ}{'detail'} .= "\L$proc: $cost\E\n"; + $report_info{full_function_details}{"\L$proc\E"}{count} = $cost; + foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { + next if (!$cost_detail{$d}); + $report_info{full_function_details}{"\L$proc\E"}{info} .= "\t$d => $cost_detail{$d}"; + $report_info{full_function_details}{"\L$proc\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); + $report_info{full_function_details}{"\L$proc\E"}{info} .= "\n"; + push(@{$report_info{full_function_details}{"\L$proc\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); + } + } + } + $report_info{'Objects'}{$typ}{'comment'} = "Total size of procedure code: $total_size bytes."; + } + elsif ($typ eq 'PACKAGE BODY') + { + $self->{packages} = $self->_get_packages(); + my $total_size = 0; + my $number_fct = 0; + my $number_pkg = 0; + foreach my $pkg (sort keys %{$self->{packages}}) { + next if (!$self->{packages}{$pkg}{text}); + $number_pkg++; + $total_size += length($self->{packages}{$pkg}{text}); + # Remove comment and text constant, they are not useful in assessment + $self->_remove_comments(\$self->{packages}{$pkg}{text}); + $self->{comment_values} = (); + $self->{text_values} = (); + my @codes = split(/CREATE(?: OR REPLACE)?(?: EDITIONABLE| NONEDITIONABLE)? PACKAGE\s+/i, $self->{packages}{$pkg}{text}); + foreach my $txt (@codes) { + next if ($txt !~ /^BODY\s+/is); + my %infos = $self->_lookup_package("CREATE OR REPLACE PACKAGE $txt"); + foreach my $f (sort keys %infos) { + next if (!$f); + if ($self->{estimate_cost}) { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $infos{$f}{code}); + $report_info{'Objects'}{$typ}{'cost_value'} += $cost; + $report_info{'Objects'}{$typ}{'detail'} .= "\L$f: $cost\E\n"; + $report_info{full_function_details}{"\L$f\E"}{count} = $cost; + foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) { + next if (!$cost_detail{$d}); + $report_info{full_function_details}{"\L$f\E"}{info} .= "\t$d => $cost_detail{$d}"; + $report_info{full_function_details}{"\L$f\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); + $report_info{full_function_details}{"\L$f\E"}{info} .= "\n"; + push(@{$report_info{full_function_details}{"\L$f\E"}{keywords}}, $d) if (($d ne 'SIZE') && ($d ne 'TEST')); + } + } + $number_fct++; + } + } + } + $self->{packages} = (); + if ($self->{estimate_cost}) { + $report_info{'Objects'}{$typ}{'cost_value'} += ($number_pkg*$Ora2Pg::PLSQL::OBJECT_SCORE{'PACKAGE BODY'}); + } + $report_info{'Objects'}{$typ}{'comment'} = "Total size of package code: $total_size bytes. Number of procedures and functions found inside those packages: $number_fct."; + } + elsif ( ($typ eq 'SYNONYM') && !$self->{is_mysql} ) + { + foreach my $t (sort {$a cmp $b} keys %synonyms) { + if ($synonyms{$t}{dblink}) { + $report_info{'Objects'}{$typ}{'detail'} .= "\L$synonyms{$t}{owner}.$t\E is a link to \L$synonyms{$t}{table_owner}.$synonyms{$t}{table_name}\@$synonyms{$t}{dblink}\E\n"; + } else { + $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E is an alias to $synonyms{$t}{table_owner}.$synonyms{$t}{table_name}\n"; + } + } + $report_info{'Objects'}{$typ}{'comment'} = "SYNONYMs will be exported as views. SYNONYMs do not exists with PostgreSQL but a common workaround is to use views or set the PostgreSQL search_path in your session to access object outside the current schema."; + } + elsif ($typ eq 'INDEX PARTITION') + { + $report_info{'Objects'}{$typ}{'comment'} = "Only local indexes partition are exported, they are build on the column used for the partitioning."; + } + elsif ($typ eq 'TABLE PARTITION') + { + my %partitions = $self->_get_partitions_list(); + foreach my $t (sort keys %partitions) { + $report_info{'Objects'}{$typ}{'detail'} .= " $partitions{$t} $t partitions.\n"; + } + $report_info{'Objects'}{$typ}{'comment'} = "Partitions are exported using table inheritance and check constraint. Hash and Key partitions are not supported by PostgreSQL and will not be exported."; + } + elsif ($typ eq 'GLOBAL TEMPORARY TABLE') + { + $report_info{'Objects'}{$typ}{'comment'} = "Global temporary table are not supported by PostgreSQL and will not be exported. You will have to rewrite some application code to match the PostgreSQL temporary table behavior."; + foreach my $t (sort keys %global_tables) { + $report_info{'Objects'}{$typ}{'detail'} .= "\L$t\E\n"; + } + } + elsif ($typ eq 'CLUSTER') + { + $report_info{'Objects'}{$typ}{'comment'} = "Clusters are not supported by PostgreSQL and will not be exported."; + } + elsif ($typ eq 'VIEW') + { + if ($self->{estimate_cost}) + { + foreach my $view (sort keys %view_infos) + { + # Remove unsupported definitions from the ddl statement + $view_infos{$view}{text} =~ s/\s*WITH\s+READ\s+ONLY//is; + $view_infos{$view}{text} =~ s/\s*OF\s+([^\s]+)\s+(WITH|UNDER)\s+[^\)]+\)//is; + $view_infos{$view}{text} =~ s/\s*OF\s+XMLTYPE\s+[^\)]+\)//is; + $view_infos{$view}{text} = $self->_format_view($view, $view_infos{$view}{text}); + + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $view_infos{$view}{text}, 'VIEW'); + $report_info{'Objects'}{$typ}{'cost_value'} += $cost; + # Do not show view that just have to be tested + next if (!$cost); + $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'VIEW'}; + # Show detail about views that might need manual rewritting + $report_info{'Objects'}{$typ}{'detail'} .= "\L$view: $cost\E\n"; + $report_info{full_view_details}{"\L$view\E"}{count} = $cost; + foreach my $d (sort { $cost_detail{$b} <=> $cost_detail{$a} } keys %cost_detail) + { + next if (!$cost_detail{$d}); + $report_info{full_view_details}{"\L$view\E"}{info} .= "\t$d => $cost_detail{$d}"; + $report_info{full_view_details}{"\L$view\E"}{info} .= " (cost: ${$uncovered_score}{$d})" if (${$uncovered_score}{$d}); + $report_info{full_view_details}{"\L$view\E"}{info} .= "\n"; + push(@{$report_info{full_view_details}{"\L$view\E"}{keywords}}, $d); + } + } + } + $report_info{'Objects'}{$typ}{'comment'} = "Views are fully supported but can use specific functions."; + } + elsif ($typ eq 'DATABASE LINK') + { + my $def_fdw = 'oracle_fdw'; + $def_fdw = 'mysql_fdw' if ($self->{is_mysql}); + $report_info{'Objects'}{$typ}{'comment'} = "Database links will be exported as SQL/MED PostgreSQL's Foreign Data Wrapper (FDW) extensions using $def_fdw."; + if ($self->{estimate_cost}) { + $report_info{'Objects'}{$typ}{'cost_value'} = ($Ora2Pg::PLSQL::OBJECT_SCORE{'DATABASE LINK'}*$objects{$typ}); + } + } + elsif ($typ eq 'JOB') + { + $report_info{'Objects'}{$typ}{'comment'} = "Job are not exported. You may set external cron job with them."; + if ($self->{estimate_cost}) { + $report_info{'Objects'}{$typ}{'cost_value'} = ($Ora2Pg::PLSQL::OBJECT_SCORE{'JOB'}*$objects{$typ}); + } + } + $report_info{'total_cost_value'} += $report_info{'Objects'}{$typ}{'cost_value'}; + $report_info{'Objects'}{$typ}{'cost_value'} = sprintf("%2.2f", $report_info{'Objects'}{$typ}{'cost_value'}); + } + + if (!$self->{quiet} && !$self->{debug}) + { + print STDERR $self->progress_bar($idx, $num_total_obj, 25, '=', 'objects types', 'end of objects auditing.'), "\n"; + } + + # DBA_AUDIT_TRAIL queries will not be count if no audit user is give + if ($self->{audit_user}) + { + my $tbname = 'DBA_AUDIT_TRAIL'; + $tbname = 'general_log' if ($self->{is_mysql}); + $report_info{'Objects'}{'QUERY'}{'number'} = 0; + $report_info{'Objects'}{'QUERY'}{'invalid'} = 0; + $report_info{'Objects'}{'QUERY'}{'comment'} = "Normalized queries found in $tbname for user(s): $self->{audit_user}"; + my %queries = $self->_get_audit_queries(); + foreach my $q (sort {$a <=> $b} keys %queries) { + $report_info{'Objects'}{'QUERY'}{'number'}++; + my $sql_q = Ora2Pg::PLSQL::convert_plsql_code($self, $queries{$q}); + if ($self->{estimate_cost}) { + my ($cost, %cost_detail) = Ora2Pg::PLSQL::estimate_cost($self, $sql_q, 'QUERY'); + $cost += $Ora2Pg::PLSQL::OBJECT_SCORE{'QUERY'}; + $report_info{'Objects'}{'QUERY'}{'cost_value'} += $cost; + $report_info{'total_cost_value'} += $cost; + } + } + $report_info{'Objects'}{'QUERY'}{'cost_value'} = sprintf("%2.2f", $report_info{'Objects'}{'QUERY'}{'cost_value'}); + } + $report_info{'total_cost_value'} = sprintf("%2.2f", $report_info{'total_cost_value'}); + + # Display report in the requested format + $self->_show_report(%report_info); + + } + elsif ($type eq 'SHOW_SCHEMA') + { + # Get all tables information specified by the DBI method table_info + $self->logit("Showing all schema...\n", 1); + my $sth = $self->_schema_list() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + my $warning = ''; + my $ret = $self->is_reserved_words($row[0]); + if ($ret == 1) { + $warning = " (Warning: '$row[0]' is a reserved word in PostgreSQL)"; + } elsif ($ret == 2) { + $warning = " (Warning: '$row[0]' object name with numbers only must be double quoted in PostgreSQL)"; + } + if (!$self->{is_mysql}) { + $self->logit("SCHEMA $row[0]$warning\n", 0); + } else { + $self->logit("DATABASE $row[0]$warning\n", 0); + } + } + $sth->finish(); + } + elsif ( ($type eq 'SHOW_TABLE') || ($type eq 'SHOW_COLUMN') ) + { + + # Get all tables information specified by the DBI method table_info + $self->logit("Showing table information...\n", 1); + + # Retrieve tables informations + my %tables_infos = $self->_table_info(); + + # Retrieve column identity information + $self->logit("Retrieving column identity information...\n", 1); + %{ $self->{identity_info} } = $self->_get_identities(); + + # Retrieve all columns information + my %columns_infos = (); + if ($type eq 'SHOW_COLUMN') + { + %columns_infos = $self->_column_info('',$self->{schema}, 'TABLE'); + foreach my $tb (keys %columns_infos) { + foreach my $c (keys %{$columns_infos{$tb}}) { + push(@{$self->{tables}{$tb}{column_info}{$c}}, @{$columns_infos{$tb}{$c}}); + } + } + %columns_infos = (); + + # Look for encrypted columns + %{$self->{encrypted_column}} = $self->_encrypted_columns('',$self->{schema}); + + # Retrieve index informations + my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('',$self->{schema}); + foreach my $tb (keys %{$indexes}) { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{indexes}} = %{$indexes->{$tb}}; + } + foreach my $tb (keys %{$idx_type}) { + next if (!exists $tables_infos{$tb}); + %{$self->{tables}{$tb}{idx_type}} = %{$idx_type->{$tb}}; + } + } + + # Get partition list to mark tables with partition. + $self->logit("Looking to subpartition information...\n", 1); + my %subpartitions_list = $self->_get_subpartitioned_table(); + $self->logit("Looking to partitioned tables information...\n", 1); + my %partitions = $self->_get_partitioned_table(%subpartitions_list); + + # Look for external tables + my %externals = (); + if (!$self->{is_mysql} && ($self->{db_version} !~ /Release 8/)) { + $self->logit("Looking to external tables information...\n", 1); + %externals = $self->_get_external_tables(); + } + + # Ordering tables by name by default + my @ordered_tables = sort { $a cmp $b } keys %tables_infos; + if (lc($self->{data_export_order}) eq 'size') + { + @ordered_tables = sort { + ($tables_infos{$b}{num_rows} || $tables_infos{$a}{num_rows}) ? + $tables_infos{$b}{num_rows} <=> $tables_infos{$a}{num_rows} : + $a cmp $b + } keys %tables_infos; + } + + my @done = (); + my $id = 0; + # Set the table information for each class found + my $i = 1; + my $total_row_num = 0; + foreach my $t (@ordered_tables) + { + # Jump to desired extraction + if (grep(/^$t$/, @done)) { + $self->logit("Duplicate entry found: $t\n", 1); + next; + } else { + push(@done, $t); + } + my $warning = ''; + + # Set the number of partition if any + if (exists $partitions{"\L$t\E"}) { + my $upto = ''; + $upto = 'up to ' if ($partitions{"\L$t\E"}{count} == 1048575); + $warning .= " - $upto" . $partitions{"\L$t\E"}{count} . " " . $partitions{"\L$t\E"}{type} . " partitions"; + $warning .= " with subpartitions" if ($partitions{"\L$t\E"}{composite}); + } + + # Search for reserved keywords + my $ret = $self->is_reserved_words($t); + if ($ret == 1) { + $warning .= " (Warning: '$t' is a reserved word in PostgreSQL)"; + } elsif ($ret == 2) { + $warning .= " (Warning: '$t' object name with numbers only must be double quoted in PostgreSQL)"; + } + + $total_row_num += $tables_infos{$t}{num_rows}; + + # Show table information + my $kind = ''; + $kind = ' FOREIGN' if ($tables_infos{$t}{connection}); + if ($tables_infos{$t}{partitioned}) { + $kind = ' PARTITIONED'; + } + if (exists $externals{$t}) { + $kind = ' EXTERNAL'; + } + if ($tables_infos{$t}{nologging}) { + $kind .= ' UNLOGGED'; + } + my $tname = $t; + if (!$self->{is_mysql}) { + $tname = "$tables_infos{$t}{owner}.$t" if ($self->{debug}); + $self->logit("[$i]$kind TABLE $tname (owner: $tables_infos{$t}{owner}, $tables_infos{$t}{num_rows} rows)$warning\n", 0); + } else { + $self->logit("[$i]$kind TABLE $tname ($tables_infos{$t}{num_rows} rows)$warning\n", 0); + } + + # Set the fields information + if ($type eq 'SHOW_COLUMN') + { + # Collect column's details for the current table with attempt to preserve column declaration order + foreach my $k (sort { + if (!$self->{reordering_columns}) { + $self->{tables}{$t}{column_info}{$a}[11] <=> $self->{tables}{$t}{column_info}{$b}[11]; + } else { + my $tmpa = $self->{tables}{$t}{column_info}{$a}; + $tmpa->[2] =~ s/\D//g; + my $typa = $self->_sql_type($tmpa->[1], $tmpa->[2], $tmpa->[5], $tmpa->[6], $tmpa->[4]); + $typa =~ s/\(.*//; + my $tmpb = $self->{tables}{$t}{column_info}{$b}; + $tmpb->[2] =~ s/\D//g; + my $typb = $self->_sql_type($tmpb->[1], $tmpb->[2], $tmpb->[5], $tmpb->[6], $tmpb->[4]); + $typb =~ s/\(.*//; + $TYPALIGN{$typb} <=> $TYPALIGN{$typa}; + } + } keys %{$self->{tables}{$t}{column_info}}) + { + # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,SRID,SDO_DIM,SDO_GTYPE + my $d = $self->{tables}{$t}{column_info}{$k}; + $d->[2] =~ s/\D//g; + my $type1 = $self->_sql_type($d->[1], $d->[2], $d->[5], $d->[6], $d->[4]); + $type1 = "$d->[1], $d->[2]" if (!$type1); + + # Check if we need auto increment + $warning = ''; + if ($d->[12] eq 'auto_increment' || $d->[12] eq '1') + { + if ($type1 !~ s/bigint/bigserial/) + { + if ($type1 !~ s/smallint/smallserial/) { + $type1 =~ s/integer/serial/; + } + } + if ($type1 =~ /serial/) { + $warning = " - Seq last value: $tables_infos{$t}{auto_increment}"; + } + } + $type1 = $self->{'modify_type'}{"\L$t\E"}{"\L$k\E"} if (exists $self->{'modify_type'}{"\L$t\E"}{"\L$k\E"}); + my $align = ''; + my $len = $d->[2]; + if (($d->[1] =~ /char/i) && ($d->[7] > $d->[2])) { + $d->[2] = $d->[7]; + } + $self->logit("\t$d->[0] : $d->[1]"); + if ($d->[1] !~ /SDO_GEOMETRY/) + { + if ($d->[2] && !$d->[5]) { + $self->logit("($d->[2])"); + } + elsif ($d->[5] && ($d->[1] =~ /NUMBER/i) ) + { + $self->logit("($d->[5]"); + $self->logit(",$d->[6]") if ($d->[6]); + $self->logit(")"); + } + if ($self->{reordering_columns}) + { + my $typ = $type1; + $typ =~ s/\(.*//; + $align = " - typalign: $TYPALIGN{$typ}"; + } + } + else + { + # 12:SRID,13:SDO_DIM,14:SDO_GTYPE + # Set the dimension, array is (srid, dims, gtype) + my $suffix = ''; + if ($d->[13] == 3) { + $suffix = 'Z'; + } elsif ($d->[13] == 4) { + $suffix = 'ZM'; + } + my $gtypes = ''; + if (!$d->[14] || ($d->[14] =~ /,/) ) { + $gtypes = $ORA2PG_SDO_GTYPE{0}; + } else { + $gtypes = $d->[14]; + } + $type1 = "geometry($gtypes$suffix"; + if ($d->[12]) { + $type1 .= ",$d->[12]"; + } + $type1 .= ")"; + $type1 .= " - $d->[14]" if ($d->[14] =~ /,/); + + } + my $ret = $self->is_reserved_words($d->[0]); + if ($ret == 1) { + $warning .= " (Warning: '$d->[0]' is a reserved word in PostgreSQL)"; + } elsif ($ret == 2) { + $warning .= " (Warning: '$d->[0]' object name with numbers only must be double quoted in PostgreSQL)"; + } elsif ($ret == 3) { + $warning = " (Warning: '$d->[0]' is a system column in PostgreSQL)"; + } + # Check if this column should be replaced by a boolean following table/column name + my $typlen = $d->[5]; + $typlen ||= $d->[2]; + if (grep(/^$d->[0]$/i, @{$self->{'replace_as_boolean'}{uc($t)}})) { + $type1 = 'boolean'; + # Check if this column should be replaced by a boolean following type/precision + } elsif (exists $self->{'replace_as_boolean'}{uc($d->[1])} && ($self->{'replace_as_boolean'}{uc($d->[1])}[0] == $typlen)) { + $type1 = 'boolean'; + } + + # Autoincremented columns + if (!$self->{schema} && $self->{export_schema}) { + $d->[8] = "$d->[9].$d->[8]"; + } + if (exists $self->{identity_info}{$d->[8]}{$d->[0]}) + { + if ($self->{pg_supports_identity}) + { + $type1 = 'bigint'; # Force bigint + $type1 .= " GENERATED $self->{identity_info}{$d->[8]}{$d->[0]}{generation} AS IDENTITY"; + $type1 .= " (" . $self->{identity_info}{$d->[8]}{$d->[0]}{options} . ')' if (exists $self->{identity_info}{$d->[8]}{$d->[0]}{options} && $self->{identity_info}{$d->[8]}{$d->[0]}{options} ne ''); + } + else + { + $type1 =~ s/bigint$/bigserial/; + $type1 =~ s/smallint/smallserial/; + $type1 =~ s/(integer|int)$/serial/; + } + } + + my $encrypted = ''; + $encrypted = " [encrypted]" if (exists $self->{encrypted_column}{"$t.$k"}); + my $virtual = ''; + $virtual = " [virtual column]" if ($d->[10] eq 'YES'); + $self->logit(" => $type1$warning$align$virtual$encrypted\n"); + } + } + $i++; + } + $self->logit("----------------------------------------------------------\n", 0); + $self->logit("Total number of rows: $total_row_num\n\n", 0); + $self->logit("Top $self->{top_max} of tables sorted by number of rows:\n", 0); + $i = 1; + foreach my $t (sort {$tables_infos{$b}{num_rows} <=> $tables_infos{$a}{num_rows}} keys %tables_infos) { + my $tname = $t; + if (!$self->{is_mysql}) { + $tname = "$tables_infos{$t}{owner}.$t" if ($self->{debug}); + } + $self->logit("\t[$i] TABLE $tname has $tables_infos{$t}{num_rows} rows\n", 0); + $i++; + last if ($i > $self->{top_max}); + } + $self->logit("Top $self->{top_max} of largest tables:\n", 0); + $i = 1; + if (!$self->{is_mysql}) { + my %largest_table = $self->_get_largest_tables(); + foreach my $t (sort { $largest_table{$b} <=> $largest_table{$a} } keys %largest_table) { + last if ($i > $self->{top_max}); + my $tname = $t; + $tname = "$tables_infos{$t}{owner}.$t" if ($self->{debug}); + $self->logit("\t[$i] TABLE $tname: $largest_table{$t} MB (" . $tables_infos{$t}{num_rows} . " rows)\n", 0); + $i++; + } + } else { + foreach my $t (sort {$tables_infos{$b}{size} <=> $tables_infos{$a}{size}} keys %tables_infos) { + last if ($i > $self->{top_max}); + my $tname = $t; + $self->logit("\t[$i] TABLE $tname: $tables_infos{$t}{size} MB (" . $tables_infos{$t}{num_rows} . " rows)\n", 0); + $i++; + } + } + } +} + +sub show_test_errors +{ + my ($self, $lbl_type, @errors) = @_; + + print "[ERRORS \U$lbl_type\E COUNT]\n"; + if ($#errors >= 0) { + foreach my $msg (@errors) { + print "$msg\n"; + } + } else { + if ($self->{pg_dsn}) { + print "OK, Oracle and PostgreSQL have the same number of $lbl_type.\n"; + } else { + print "No PostgreSQL connection, can not check number of $lbl_type.\n"; + } + } +} + +sub set_pg_relation_name +{ + my ($self, $table) = @_; + + my $tbmod = $self->get_replaced_tbname($table); + my $orig = ''; + $orig = " (origin: $table)" if (lc($tbmod) ne lc($table)); + my $tbname = $tbmod; + $tbname =~ s/[^"\.]+\.//; + if ($self->{pg_schema} && $self->{export_schema}) { + return ($tbmod, $orig, $self->{pg_schema}, "$self->{pg_schema}.$tbname"); + } elsif ($self->{schema} && $self->{export_schema}) { + return ($tbmod, $orig, $self->{schema}, "$self->{schema}.$tbname"); + } + + return ($tbmod, $orig, '', $tbmod); +} + +sub get_schema_condition +{ + my ($self, $attrname) = @_; + + $attrname ||= 'n.nspname'; + + if ($self->{pg_schema} && $self->{export_schema}) { + return " AND $attrname IN ('" . join("','", split(/\s*,\s*/, $self->{pg_schema})) . "')"; + } elsif ($self->{schema} && $self->{export_schema}) { + return "AND $attrname = '\L$self->{schema}\E'"; + } + + my $cond = " AND $attrname <> 'pg_catalog' AND $attrname <> 'information_schema' AND $attrname !~ '^pg_toast'"; + + return $cond; +} + + +sub _table_row_count +{ + my $self = shift; + + my $lbl = 'ORACLEDB'; + $lbl = 'MYSQL_DB' if ($self->{is_mysql}); + + # Get all tables information specified by the DBI method table_info + $self->logit("Looking for real row count in source database and PostgreSQL tables...\n", 1); + + # Retrieve tables informations + my %tables_infos = $self->_table_info($self->{count_rows}); + + #### + # Test number of row in tables + #### + my @errors = (); + print "\n"; + print "[TEST ROWS COUNT]\n"; + foreach my $t (sort keys %tables_infos) { + print "$lbl:$t:$tables_infos{$t}{num_rows}\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + my $s = $self->{dbhdest}->prepare("SELECT count(*) FROM $both;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + if (not $s->execute) { + push(@errors, "Table $both$orig does not exists in PostgreSQL database.") if ($s->state eq '42P01'); + next; + } + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $tables_infos{$t}{num_rows}) { + push(@errors, "Table $both$orig doesn't have the same number of line in source database ($tables_infos{$t}{num_rows}) and in PostgreSQL ($row[0])."); + } + last; + } + $s->finish(); + } + } + $self->show_test_errors('rows', @errors); +} + +sub _test_table +{ + my $self = shift; + + my @errors = (); + + # Get all tables information specified by the DBI method table_info + $self->logit("Looking for objects count related to source database and PostgreSQL tables...\n", 1); + + # Retrieve tables informations + my %tables_infos = $self->_table_info($self->{count_rows}); + + my $lbl = 'ORACLEDB'; + $lbl = 'MYSQL_DB' if ($self->{is_mysql}); + + #### + # Test number of index in tables + #### + print "[TEST INDEXES COUNT]\n"; + my ($uniqueness, $indexes, $idx_type, $idx_tbsp) = $self->_get_indexes('', $self->{schema}, 1); + if ($self->{is_mysql}) { + $indexes = Ora2Pg::MySQL::_count_indexes($self, '', $self->{schema}); + } + foreach my $t (keys %{$indexes}) { + next if (!exists $tables_infos{$t}); + my $numixd = scalar keys %{$indexes->{$t}}; + print "$lbl:$t:$numixd\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $schema = $self->get_schema_condition('schemaname'); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + my $s = $self->{dbhdest}->prepare("SELECT count(*) FROM pg_indexes WHERE tablename = '\L$tbmod\E'$schema;") or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + $tbmod = $1 . $tbmod if ($1); + if (not $s->execute) { + push(@errors, "Can not extract information from catalog table pg_indexes."); + next; + } + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $numixd) { + push(@errors, "Table $both$orig doesn't have the same number of indexes in source database ($numixd) and in PostgreSQL ($row[0])."); + } + last; + } + $s->finish(); + } + } + $self->show_test_errors('indexes', @errors); + @errors = (); + + #### + # Test unique constraints (excluding primary keys) + #### + print "\n"; + print "[TEST UNIQUE CONSTRAINTS COUNT]\n"; + my %unique_keys = $self->_unique_key('',$self->{schema},'U'); + my $schema_cond = $self->get_schema_condition('pg_indexes.schemaname'); + my $sql = qq{ +SELECT count(*) +FROM pg_indexes +JOIN pg_class ON (pg_class.relname=pg_indexes.indexname) +JOIN pg_constraint ON (pg_constraint.conname=pg_class.relname AND pg_constraint.connamespace=pg_class.relnamespace) +WHERE pg_indexes.tablename=? +AND pg_constraint.contype IN ('u') + $schema_cond +}; + + my $s = undef; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + foreach my $t (keys %unique_keys) { + next if (!exists $tables_infos{$t}); + my $numixd = scalar keys %{$unique_keys{$t}}; + print "$lbl:$t:$numixd\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + if (not $s->execute(lc($tbmod))) { + push(@errors, "Can not extract information from catalog about unique constraints."); + next; + } + $tbmod = $1 . $tbmod if ($1); + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $numixd) { + push(@errors, "Table $both$orig doesn't have the same number of unique constraints in source database ($numixd) and in PostgreSQL ($row[0])."); + } + last; + } + } + } + $s->finish() if ($self->{pg_dsn}); + $self->show_test_errors('unique constraints', @errors); + @errors = (); + + #### + # Test primary keys only + #### + print "\n"; + print "[TEST PRIMARY KEYS COUNT]\n"; + %unique_keys = $self->_unique_key('',$self->{schema},'P'); + $schema_cond = $self->get_schema_condition('pg_indexes.schemaname'); + $sql = qq{ +SELECT count(*) +FROM pg_indexes +JOIN pg_class ON (pg_class.relname=pg_indexes.indexname) +JOIN pg_constraint ON (pg_constraint.conname=pg_class.relname AND pg_constraint.connamespace=pg_class.relnamespace) +WHERE pg_indexes.tablename=? +AND pg_constraint.contype IN ('p') + $schema_cond +}; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + foreach my $t (keys %unique_keys) { + next if (!exists $tables_infos{$t}); + my $nbpk = 0; + foreach my $c (keys %{$unique_keys{$t}}) { + $nbpk++ if ($unique_keys{$t}{$c}{type} eq 'P'); + } + print "$lbl:$t:$nbpk\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + if (not $s->execute(lc($tbmod))) { + push(@errors, "Can not extract information from catalog about primary keys."); + next; + } + $tbmod = $1 . $tbmod if ($1); + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $nbpk) { + push(@errors, "Table $both$orig doesn't have the same number of primary keys in source database ($nbpk) and in PostgreSQL ($row[0])."); + } + last; + } + } + } + $s->finish() if ($self->{pg_dsn}); + %unique_keys = (); + $self->show_test_errors('primary keys', @errors); + @errors = (); + + #### + # Test check constraints + #### + if (!$self->{is_mysql}) { + print "\n"; + print "[TEST CHECK CONSTRAINTS COUNT]\n"; + my %check_constraints = $self->_check_constraint('',$self->{schema}); + $schema_cond = $self->get_schema_condition(); + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_constraint r JOIN pg_class c ON (r.conrelid=c.oid) JOIN pg_namespace n ON (c.relnamespace=n.oid) +WHERE c.relname = ? AND r.contype = 'c' +$schema_cond +}; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + foreach my $t (keys %check_constraints) { + next if (!exists $tables_infos{$t}); + my $nbcheck = 0; + foreach my $cn (keys %{$check_constraints{$t}{constraint}}) { + $nbcheck++ if ($check_constraints{$t}{constraint}{$cn}{condition} !~ /IS NOT NULL$/); + } + print "$lbl:$t:$nbcheck\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + if (not $s->execute(lc($tbmod))) { + push(@errors, "Can not extract information from catalog about check constraints."); + next; + } + $tbmod = $1 . $tbmod if ($1); + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $nbcheck) { + push(@errors, "Table $both$orig doesn't have the same number of check constraints in source database ($nbcheck) and in PostgreSQL ($row[0])."); + } + last; + } + } + } + $s->finish() if ($self->{pg_dsn}); + %check_constraints = (); + $self->show_test_errors('check constraints', @errors); + @errors = (); + } + + #### + # Test NOT NULL constraints + #### + print "\n"; + print "[TEST NOT NULL CONSTRAINTS COUNT]\n"; + my %column_infos = $self->_column_attributes('', $self->{schema}, 'TABLE'); + $schema_cond = $self->get_schema_condition(); + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_attribute a +JOIN pg_class e ON (e.oid=a.attrelid) +JOIN pg_namespace n ON (e.relnamespace=n.oid) +WHERE e.relname = ? + AND a.attnum > 0 + AND NOT a.attisdropped AND a.attnotnull + $schema_cond +}; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + foreach my $t (keys %column_infos) { + next if (!exists $tables_infos{$t}); + my $nbnull = 0; + foreach my $cn (keys %{$column_infos{$t}}) { + if ($column_infos{$t}{$cn}{nullable} =~ /^N/) { + $nbnull++; + } + } + print "$lbl:$t:$nbnull\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + if (not $s->execute(lc($tbmod))) { + push(@errors, "Can not extract information from catalog about not null constraints."); + next; + } + $tbmod = $1 . $tbmod if ($1); + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $nbnull) { + push(@errors, "Table $both$orig doesn't have the same number of not null constraints in source database ($nbnull) and in PostgreSQL ($row[0])."); + } + last; + } + } + } + $s->finish() if ($self->{pg_dsn}); + $self->show_test_errors('not null constraints', @errors); + @errors = (); + + #### + # Test NOT NULL constraints + #### + print "\n"; + print "[TEST COLUMN DEFAULT VALUE COUNT]\n"; + $schema_cond = $self->get_schema_condition(); + $sql = qq{ +SELECT a.attname, + (SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid) for 128) + FROM pg_catalog.pg_attrdef d + WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef) +FROM pg_catalog.pg_attribute a JOIN pg_class e ON (e.oid=a.attrelid) JOIN pg_namespace n ON (e.relnamespace=n.oid) +WHERE e.relname = ? AND a.attnum > 0 AND NOT a.attisdropped + $schema_cond +}; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + my @seqs = (); + if ($self->{is_mysql}) { + @seqs = Ora2Pg::MySQL::_count_sequences($self); + } + foreach my $t (keys %column_infos) { + next if (!exists $tables_infos{$t}); + my $nbdefault = 0; + foreach my $cn (keys %{$column_infos{$t}}) { + if ($column_infos{$t}{$cn}{default} ne '' && uc($column_infos{$t}{$cn}{default}) ne 'NULL') { + $nbdefault++; + } + } + if (grep(/^$t$/i, @seqs)) { + $nbdefault++; + } + print "$lbl:$t:$nbdefault\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + if (not $s->execute(lc($tbmod))) { + push(@errors, "Can not extract information from catalog about column default value."); + next; + } + $tbmod = $1 . $tbmod if ($1); + my $pgdef = 0; + while ( my @row = $s->fetchrow()) { + $pgdef++ if ($row[1] ne ''); + } + print "POSTGRES:$both$orig:$pgdef\n"; + if ($pgdef != $nbdefault) { + push(@errors, "Table $both$orig doesn't have the same number of column default value in source database ($nbdefault) and in PostgreSQL ($pgdef)."); + } + } + } + $s->finish() if ($self->{pg_dsn}); + %column_infos = (); + $self->show_test_errors('column default value', @errors); + @errors = (); + + #### + # Test foreign keys + #### + print "\n"; + print "[TEST FOREIGN KEYS COUNT]\n"; + my ($foreign_link, $foreign_key) = $self->_foreign_key('',$self->{schema}); + $schema_cond = $self->get_schema_condition(); + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_constraint r JOIN pg_class c ON (r.conrelid=c.oid) JOIN pg_namespace n ON (c.relnamespace=n.oid) +WHERE c.relname = ? + AND r.contype = 'f' + $schema_cond +}; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + foreach my $t (keys %{$foreign_link}) { + next if (!exists $tables_infos{$t}); + my $nbfk = scalar keys %{$foreign_link->{$t}}; + print "$lbl:$t:$nbfk\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + if (not $s->execute(lc($tbmod))) { + push(@errors, "Can not extract information from catalog about foreign key constraints."); + next; + } + $tbmod = $1 . $tbmod if ($1); + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $nbfk) { + push(@errors, "Table $both$orig doesn't have the same number of foreign key constraints in source database ($nbfk) and in PostgreSQL ($row[0])."); + } + last; + } + } + } + $s->finish() if ($self->{pg_dsn}); + $self->show_test_errors('foreign keys', @errors); + @errors = (); + + #### + # Test triggers + #### + print "\n"; + print "[TEST TABLE TRIGGERS COUNT]\n"; + my %triggers = $self->_list_triggers(); + $schema_cond = $self->get_schema_condition(); + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_trigger t JOIN pg_class c ON (t.tgrelid=c.oid) JOIN pg_namespace n ON (c.relnamespace=n.oid) +WHERE c.relname = ? + AND (NOT t.tgisinternal OR (t.tgisinternal AND t.tgenabled = 'D')) + $schema_cond +}; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + } + foreach my $t (keys %triggers) { + next if (!exists $tables_infos{$t}); + my $nbtrg = $#{$triggers{$t}}+1; + print "$lbl:$t:$nbtrg\n"; + if ($self->{pg_dsn}) { + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + $tbmod =~ s/^([^\.]+\.)//; # Remove schema part from table name + if (not $s->execute(lc($tbmod))) { + push(@errors, "Can not extract information from catalog about foreign key constraints."); + next; + } + $tbmod = $1 . $tbmod if ($1); + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$both$orig:$row[0]\n"; + if ($row[0] != $nbtrg) { + push(@errors, "Table $both$orig doesn't have the same number of triggers in source database ($nbtrg) and in PostgreSQL ($row[0])."); + } + last; + } + } + } + $s->finish() if ($self->{pg_dsn}); + $self->show_test_errors('table triggers', @errors); + @errors = (); + + #### + # Test partitions + #### + print "\n"; + print "[TEST PARTITION COUNT]\n"; + my %partitions = $self->_get_partitioned_table(); + $schema_cond = $self->get_schema_condition('nmsp_parent.nspname'); + $schema_cond =~ s/^ AND/ WHERE/; + $sql = qq{ +SELECT + nmsp_parent.nspname AS parent_schema, + parent.relname AS parent, + COUNT(*) +FROM pg_inherits + JOIN pg_class parent ON pg_inherits.inhparent = parent.oid + JOIN pg_class child ON pg_inherits.inhrelid = child.oid + JOIN pg_namespace nmsp_parent ON nmsp_parent.oid = parent.relnamespace + JOIN pg_namespace nmsp_child ON nmsp_child.oid = child.relnamespace +$schema_cond +GROUP BY + parent_schema, + parent; +}; + my %pg_part = (); + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + if (not $s->execute()) { + push(@errors, "Can not extract information from catalog about PARTITION."); + next; + } + while ( my @row = $s->fetchrow()) { + $pg_part{$row[1]} = $row[2]; + } + $s->finish(); + } + foreach my $t (keys %partitions) { + next if (!exists $tables_infos{$t}); + print "$lbl:$t:", $partitions{"\L$t\E"}{count}, "\n"; + my ($tbmod, $orig, $schema, $both) = $self->set_pg_relation_name($t); + if (exists $pg_part{$tbmod}) { + print "POSTGRES:$both$orig:$pg_part{$tbmod}\n"; + if ($pg_part{$tbmod} != $partitions{"\L$t\E"}{count}) { + push(@errors, "Table $both$orig doesn't have the same number of partitions in source database (" . $partitions{"\L$t\E"}{count} . ") and in PostgreSQL ($pg_part{$tbmod})."); + } + } else { + push(@errors, "Table $both$orig doesn't have the same number of partitions in source database (" . $partitions{"\L$t\E"}{count} . ") and in PostgreSQL (0)."); + } + } + $self->show_test_errors('PARTITION', @errors); + @errors = (); + + print "\n"; + print "[TEST TABLE COUNT]\n"; + my $nbobj = scalar keys %tables_infos; + $schema_cond = $self->get_schema_condition(); + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_class c + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE c.relkind IN ('r','') + $schema_cond +}; + + print "$lbl:TABLE:$nbobj\n"; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + if (not $s->execute()) { + push(@errors, "Can not extract information from catalog about $obj_type."); + next; + } + while ( my @row = $s->fetchrow()) { + print "POSTGRES:TABLE:$row[0]\n"; + if ($row[0] != $nbobj) { + push(@errors, "TABLE does not have the same count in source database ($nbobj) and in PostgreSQL ($row[0])."); + } + last; + } + $s->finish(); + } + $self->show_test_errors('TABLE', @errors); + @errors = (); + + print "\n"; + print "[TEST TRIGGER COUNT]\n"; + $nbobj = 0; + foreach my $t (keys %triggers) { + next if (!exists $tables_infos{$t}); + $nbobj += $#{$triggers{$t}}+1; + } + $schema_cond = $self->get_schema_condition(); + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_trigger t JOIN pg_class c ON (c.oid = t.tgrelid) JOIN pg_namespace n ON (c.relnamespace=n.oid) +WHERE (NOT t.tgisinternal OR (t.tgisinternal AND t.tgenabled = 'D')) + $schema_cond +}; + + print "$lbl:TRIGGER:$nbobj\n"; + if ($self->{pg_dsn}) { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + if (not $s->execute()) { + push(@errors, "Can not extract information from catalog about $obj_type."); + next; + } + while ( my @row = $s->fetchrow()) { + print "POSTGRES:TRIGGER:$row[0]\n"; + if ($row[0] != $nbobj) { + push(@errors, "TRIGGER does not have the same count in source database ($nbobj) and in PostgreSQL ($row[0])."); + } + last; + } + $s->finish(); + } + $self->show_test_errors('TRIGGER', @errors); + @errors = (); + +} + +sub _unitary_test_views +{ + my $self = shift; + + # Get all tables information specified by the DBI method table_info + $self->logit("Unitary test of views between source database and PostgreSQL...\n", 1); + + # First of all extract all views from PostgreSQL database + my $schema_clause = $self->get_schema_condition(); + my $sql = qq{ +SELECT c.relname,n.nspname +FROM pg_catalog.pg_class c + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE c.relkind IN ('v','') + $schema_clause +}; + my %list_views = (); + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + if (not $s->execute()) { + push(@errors, "Can not extract information from catalog about views."); + next; + } + while ( my @row = $s->fetchrow()) { + $list_views{$row[0]} = $row[1]; + } + $s->finish(); + } + + my $lbl = 'ORACLEDB'; + $lbl = 'MYSQL_DB' if ($self->{is_mysql}); + + print "[UNITARY TEST OF VIEWS]\n"; + foreach my $v (sort keys %list_views) { + # Execute init settings if any + # Count rows returned by all view on the source database + $sql = "SELECT count(*) FROM $v"; + my $sth = $self->{dbh}->prepare($sql) or $self->logit("ERROR: " . $self->{dbh}->errstr . "\n", 0, 0); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 0); + my @row = $sth->fetchrow(); + my $ora_ct = $row[0]; + print "$lbl:$v:", join('|', @row), "\n"; + $sth->finish; + # Execute view in the PostgreSQL database + $sql = "SELECT count(*) FROM $v;"; + $sth = $self->{dbhdest}->prepare($sql) or $self->logit("ERROR: " . $self->{dbhdest}->errstr . "\n", 0, 0); + $sth->execute or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 0); + @row = $sth->fetchrow(); + $sth->finish; + my $pg_ct = $row[0]; + print "POSTGRES:$v:", join('|', @row), "\n"; + if ($pg_ct != $ora_ct) { + print "ERROR: view $v returns different row count [oracle: $ora_ct, postgresql: $pg_ct]\n"; + } + } +} + +sub _count_object +{ + my $self = shift; + my $obj_type = shift; + + # Get all tables information specified by the DBI method table_info + $self->logit("Looking for source database and PostgreSQL objects count...\n", 1); + + my $lbl = 'ORACLEDB'; + $lbl = 'MYSQL_DB' if ($self->{is_mysql}); + + my $schema_clause = $self->get_schema_condition(); + my $nbobj = 0; + my $sql = ''; + if ($obj_type eq 'VIEW') { + my %obj_infos = $self->_get_views(); + $nbobj = scalar keys %obj_infos; + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_class c + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE c.relkind IN ('v','') + $schema_clause +}; + } elsif ($obj_type eq 'MVIEW') { + my %obj_infos = $self->_get_materialized_views(); + $nbobj = scalar keys %obj_infos; + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_class c + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE c.relkind IN ('m','') + $schema_clause +}; + } elsif ($obj_type eq 'SEQUENCE') { + my $obj_infos = (); + if (!$self->{is_mysql}) { + $obj_infos = $self->_get_sequences(); + } else { + $obj_infos = Ora2Pg::MySQL::_count_sequences($self); + } + $nbobj = $#{$obj_infos} + 1; + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_class c + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE c.relkind IN ('S','') + $schema_clause +}; + } elsif ($obj_type eq 'TYPE') { + my $obj_infos = $self->_get_types(); + $nbobj = $#{$obj_infos} + 1; + $schema_clause .= " AND pg_catalog.pg_type_is_visible(t.oid)" if ($schema_clause =~ /information_schema/); + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_type t + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace +WHERE (t.typrelid = 0 OR (SELECT c.relkind = 'c' FROM pg_catalog.pg_class c WHERE c.oid = t.typrelid)) + AND NOT EXISTS(SELECT 1 FROM pg_catalog.pg_type el WHERE el.oid = t.typelem AND el.typarray = t.oid) + $schema_clause +}; + } elsif ($obj_type eq 'FDW') { + my %obj_infos = $self->_get_external_tables(); + $nbobj = scalar keys %obj_infos; + $sql = qq{ +SELECT count(*) +FROM pg_catalog.pg_class c + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE c.relkind IN ('f','') + $schema_clause +}; + } else { + return; + } + + print "\n"; + print "[TEST $obj_type COUNT]\n"; + + if ($self->{is_mysql} && ($obj_type eq 'SEQUENCE')) { + print "$lbl:AUTOINCR:$nbobj\n"; + } else { + print "$lbl:$obj_type:$nbobj\n"; + } + if ($self->{pg_dsn}) { + my $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + if (not $s->execute()) { + push(@errors, "Can not extract information from catalog about $obj_type."); + next; + } + while ( my @row = $s->fetchrow()) { + print "POSTGRES:$obj_type:$row[0]\n"; + if ($row[0] != $nbobj) { + push(@errors, "\U$obj_type\E does not have the same count in source database ($nbobj) and in PostgreSQL ($row[0])."); + } + last; + } + $s->finish(); + } + $self->show_test_errors($obj_type, @errors); + @errors = (); +} + +sub _test_function +{ + my $self = shift; + + my @errors = (); + + $self->logit("Looking for functions count related to source database and PostgreSQL functions...\n", 1); + + my $lbl = 'ORACLEDB'; + $lbl = 'MYSQL_DB' if ($self->{is_mysql}); + + #### + # Test number of function + #### + print "\n"; + print "[TEST FUNCTION COUNT]\n"; + my @fct_infos = $self->_list_all_funtions(); + my $schema_clause = " AND n.nspname NOT IN ('pg_catalog','information_schema')"; + $sql = qq{ +SELECT n.nspname,proname,prorettype +FROM pg_catalog.pg_proc p + LEFT JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace + LEFT JOIN pg_catalog.pg_type t ON t.oid=p.prorettype +WHERE t.typname <> 'trigger' +$schema_clause +}; + + my $nbobj = $#fct_infos + 1; + print "$lbl:FUNCTION:$nbobj\n"; + if ($self->{pg_dsn}) + { + $s = $self->{dbhdest}->prepare($sql) or $self->logit("FATAL: " . $self->{dbhdest}->errstr . "\n", 0, 1); + if (not $s->execute()) { + push(@errors, "Can not extract information from catalog about $obj_type."); + next; + } + my $pgfct = 0; + my %pg_function = (); + while ( my @row = $s->fetchrow()) + { + $pgfct++; + my $fname = $row[1]; + if ($row[0] ne 'public') { + $fname = $row[0] . '.' . $row[1]; + } + $pg_function{lc($fname)} = 1; + } + print "POSTGRES:FUNCTION:$pgfct\n"; + if ($pgfct != $nbobj) { + push(@errors, "FUNCTION does not have the same count in source database ($nbobj) and in PostgreSQL ($pgfct)."); + } + $s->finish(); + # search for missing funtion + foreach my $f (@fct_infos) + { + my $found = 0; + foreach my $pgf (keys %pg_function) + { + $found = 1, last if (lc($f) eq lc($pgf)); + if ($f !~ /\./) { + $found = 1, last if ($pgf =~ /^[^\.]+\.$f$/i); + } else { + $found = 1, last if ($pgf =~ /^$f$/i); + } + } + push(@errors, "Function $f is missing in PostgreSQL database.") if (!$found); + } + } + $self->show_test_errors('FUNCTION', @errors); + @errors = (); + print "\n"; +} + +=head2 _get_version + +This function retrieves the Oracle version information + +=cut + +sub _get_version +{ + my $self = shift; + + return Ora2Pg::MySQL::_get_version($self) if ($self->{is_mysql}); + + my $oraver = ''; + my $sql = "SELECT BANNER FROM v\$version"; + + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + $oraver = $row[0]; + last; + } + $sth->finish(); + + chomp($oraver); + $oraver =~ s/ \- .*//; + + return $oraver; +} + +=head2 _get_database_size + +This function retrieves the size of the Oracle database in MB + +=cut + +sub _get_database_size +{ + my $self = shift; + + return Ora2Pg::MySQL::_get_database_size($self) if ($self->{is_mysql}); + + my $mb_size = ''; + my $sql = "SELECT sum(bytes)/1024/1024 FROM USER_SEGMENTS"; + if (!$self->{user_grants}) { + $sql = "SELECT sum(bytes)/1024/1024 FROM DBA_SEGMENTS"; + if ($self->{schema}) { + $sql .= " WHERE OWNER='$self->{schema}' "; + } else { + $sql .= " WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + } + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + $mb_size = sprintf("%.2f MB", $row[0]); + last; + } + $sth->finish(); + + return $mb_size; +} + +=head2 _get_objects + +This function retrieves all object the Oracle information +except SYNONYM and temporary objects + +=cut + +sub _get_objects +{ + my $self = shift; + + return Ora2Pg::MySQL::_get_objects($self) if ($self->{is_mysql}); + + my $oraver = ''; + # OWNER|OBJECT_NAME|SUBOBJECT_NAME|OBJECT_ID|DATA_OBJECT_ID|OBJECT_TYPE|CREATED|LAST_DDL_TIME|TIMESTAMP|STATUS|TEMPORARY|GENERATED|SECONDARY + my $sql = "SELECT OBJECT_NAME,OBJECT_TYPE,STATUS FROM $self->{prefix}_OBJECTS WHERE TEMPORARY='N' AND GENERATED='N' AND SECONDARY='N' AND OBJECT_TYPE <> 'SYNONYM'"; + if ($self->{schema}) { + $sql .= " AND OWNER='$self->{schema}'"; + } else { + $sql .= " AND OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + my @infos = (); + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + push(@infos, join('|', @{$sth->{NAME}})); + $sth->execute or return undef; + my %count = (); + while ( my @row = $sth->fetchrow()) + { + my $valid = ($row[2] eq 'VALID') ? 0 : 1; + push(@{$infos{$row[1]}}, { ( name => $row[0], invalid => $valid ) }); + $count{$row[1]}{$valid}++; + } + $sth->finish(); + + if ($self->{debug}) + { + foreach my $k (sort keys %count) + { + print STDERR "\tFound $count{$k}{0} valid and ", ($count{$k}{1}||0), " invalid object $k\n"; + } + } + + return %infos; +} + +sub _list_all_funtions +{ + my $self = shift; + + return Ora2Pg::MySQL::_list_all_funtions($self) if ($self->{is_mysql}); + + my $oraver = ''; + # OWNER|OBJECT_NAME|PROCEDURE_NAME|OBJECT_TYPE + my $sql = qq{ +SELECT p.owner,p.object_name,p.procedure_name,o.object_type + FROM $self->{prefix}_PROCEDURES p + JOIN $self->{prefix}_OBJECTS o ON p.owner = o.owner + AND p.object_name = o.object_name + WHERE o.object_type IN ('PROCEDURE','PACKAGE','FUNCTION') + AND o.TEMPORARY='N' AND o.GENERATED='N' AND o.SECONDARY='N' + AND o.STATUS = 'VALID' +}; + if ($self->{db_version} =~ /Release 8/) { + $sql = qq{ +SELECT p.owner,p.object_name,p.procedure_name,o.object_type + FROM $self->{prefix}_PROCEDURES p, $self->{prefix}_OBJECTS o + WHERE o.object_type IN ('PROCEDURE','PACKAGE','FUNCTION') + AND p.owner = o.owner AND p.object_name = o.object_name + AND o.TEMPORARY='N' AND o.GENERATED='N' AND o.SECONDARY='N' + AND o.STATUS = 'VALID' +}; + } + if ($self->{schema}) { + $sql .= " AND p.OWNER='$self->{schema}'"; + } else { + $sql .= " AND p.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + my @infos = (); + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + next if (($row[3] eq 'PACKAGE') && !$row[2]); + if ( $row[2] ) { + # package_name.fct_name + push(@infos, lc("$row[1].$row[2]")); + } else { + # owner.fct_name + push(@infos, lc($row[1])); + } + } + $sth->finish(); + + return @infos; +} + +=head2 _schema_list + +This function retrieves all Oracle-native user schema. + +Returns a handle to a DB query statement. + +=cut + +sub _schema_list +{ + my $self = shift; + + return Ora2Pg::MySQL::_schema_list($self) if ($self->{is_mysql}); + + my $sql = "SELECT DISTINCT OWNER FROM ALL_TABLES WHERE OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "') ORDER BY OWNER"; + + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + $sth; +} + +=head2 _table_exists + +This function return the table name if the given table exists +else returns a empty string. + +=cut + +sub _table_exists +{ + my ($self, $schema, $table) = @_; + + return Ora2Pg::MySQL::_table_exists($self, $schema, $table) if ($self->{is_mysql}); + + my $ret = ''; + + my $sql = "SELECT TABLE_NAME FROM $self->{prefix}_TABLES WHERE OWNER = '$schema' AND TABLE_NAME = '$table'"; + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + $ret = $row[0]; + } + $sth->finish(); + return $ret; +} + + + +=head2 _get_largest_tables + +This function retrieves the list of largest table of the Oracle database in MB + +=cut + +sub _get_largest_tables +{ + my $self = shift; + + return Ora2Pg::MySQL::_get_largest_tables($self) if ($self->{is_mysql}); + + my %table_size = (); + + my $prefix = 'USER'; + my $owner_segment = ''; + $owner_segment = " AND A.OWNER='$self->{schema}'"; + if (!$self->{user_grants}) { + $prefix = 'DBA'; + $owner_segment = ' AND S.OWNER=A.OWNER'; + } + + my $sql = "SELECT * FROM ( SELECT S.SEGMENT_NAME, ROUND(S.BYTES/1024/1024) SIZE_MB FROM ${prefix}_SEGMENTS S JOIN ALL_TABLES A ON (S.SEGMENT_NAME=A.TABLE_NAME$owner_segment) WHERE S.SEGMENT_TYPE LIKE 'TABLE%' AND A.SECONDARY = 'N'"; + if ($self->{db_version} =~ /Release 8/) { + $sql = "SELECT * FROM ( SELECT A.SEGMENT_NAME, ROUND(A.BYTES/1024/1024) SIZE_MB FROM ${prefix}_SEGMENTS A WHERE A.SEGMENT_TYPE LIKE 'TABLE%'"; + } + if ($self->{db_version} !~ /Release 8/ || !$self->{user_grants}) { + if ($self->{schema}) { + $sql .= " AND A.OWNER='$self->{schema}'"; + } else { + $sql .= " AND A.OWNER NOT IN ('" . join("','", @{$self->{sysusers}}) . "')"; + } + } + if ($self->{db_version} =~ /Release 8/) { + $sql .= $self->limit_to_objects('TABLE', 'A.SEGMENT_NAME'); + } else { + $sql .= $self->limit_to_objects('TABLE', 'A.TABLE_NAME'); + } + + if ($self->{db_version} =~ /Release 8/) { + $sql .= " ORDER BY A.BYTES DESC, A.SEGMENT_NAME ASC) WHERE ROWNUM <= $self->{top_max}"; + } else { + $sql .= " ORDER BY S.BYTES DESC, S.SEGMENT_NAME ASC) WHERE ROWNUM <= $self->{top_max}"; + } + + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute(@{$self->{query_bind_params}}) or return undef; + while ( my @row = $sth->fetchrow()) { + $table_size{$row[0]} = $row[1]; + } + $sth->finish(); + + return %table_size; +} + + +=head2 _get_encoding + +This function retrieves the Oracle database encoding + +Returns a handle to a DB query statement. + +=cut + +sub _get_encoding +{ + my ($self, $dbh) = @_; + + my $sql = "SELECT * FROM NLS_DATABASE_PARAMETERS"; + my $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + my $language = ''; + my $territory = ''; + my $charset = ''; + my $nls_timestamp_format = ''; + my $nls_date_format = ''; + while ( my @row = $sth->fetchrow()) { + if ($row[0] eq 'NLS_LANGUAGE') { + $language = $row[1]; + } elsif ($row[0] eq 'NLS_TERRITORY') { + $territory = $row[1]; + } elsif ($row[0] eq 'NLS_CHARACTERSET') { + $charset = $row[1]; + } elsif ($row[0] eq 'NLS_TIMESTAMP_FORMAT') { + $nls_timestamp_format = $row[1]; + } elsif ($row[0] eq 'NLS_DATE_FORMAT') { + $nls_date_format = $row[1]; + } + } + $sth->finish(); + $sql = "SELECT * FROM NLS_SESSION_PARAMETERS"; + $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + my $ora_encoding = ''; + while ( my @row = $sth->fetchrow()) { + #$self->logit("SESSION PARAMETERS: $row[0] $row[1]\n", 1); + if ($row[0] eq 'NLS_LANGUAGE') { + $language = $row[1]; + } elsif ($row[0] eq 'NLS_TERRITORY') { + $territory = $row[1]; + } elsif ($row[0] eq 'NLS_TIMESTAMP_FORMAT') { + $nls_timestamp_format = $row[1]; + } elsif ($row[0] eq 'NLS_DATE_FORMAT') { + $nls_date_format = $row[1]; + } + } + $sth->finish(); + + $ora_encoding = $language . '_' . $territory . '.' . $charset; + my $pg_encoding = auto_set_encoding($charset); + + return ($ora_encoding, $charset, $pg_encoding, $nls_timestamp_format, $nls_date_format); +} + + +=head2 _compile_schema + +This function force Oracle database to compile a schema and validate or +invalidate PL/SQL code. + +When parameter $schema is the name of a schema, only this schema is recompiled +When parameter $schema is equal to 1 and SCHEMA directive is set, only this schema is recompiled +When parameter $schema is equal to 1 and SCHEMA directive is unset, all schema will be recompiled + +=cut + + +sub _compile_schema +{ + my ($self, $schema) = @_; + + my @to_compile = (); + + if ($schema and ($schema =~ /[a-z]/i)) { + push(@to_compile, $schema); + } elsif ($schema and $self->{schema}) { + push(@to_compile, $self->{schema}); + } elsif ($schema) { + my $sth = $self->_schema_list() or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@to_compile, $row[0]); + } + $sth->finish(); + } + + if ($#to_compile >= 0) { + foreach my $schm (@to_compile) { + $self->logit("Force Oracle to compile schema $schm before code extraction\n", 1); + my $sth = $self->{dbh}->do("BEGIN\nDBMS_UTILITY.compile_schema(schema => '$schm');\nEND;") + or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + } + +} + + +=head2 _datetime_format + +This function force Oracle database to format the time correctly + +=cut + +sub _datetime_format +{ + my ($self, $dbh) = @_; + + $dbh = $self->{dbh} if (!$dbh); + + if ($self->{enable_microsecond}) { + my $dim = 6; + $dim = '' if ($self->{db_version} =~ /Release [89]/); + my $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS.FF$dim'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } else { + my $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } + my $sth = $dbh->do("ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + if ($self->{enable_microsecond}) { + my $dim = 6; + $dim = '' if ($self->{db_version} =~ /Release [89]/); + $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT='YYYY-MM-DD HH24:MI:SS.FF$dim TZH:TZM'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } else { + $sth = $dbh->do("ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT='YYYY-MM-DD HH24:MI:SS TZH:TZM'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } +} + +sub _numeric_format +{ + my ($self, $dbh) = @_; + + $dbh = $self->{dbh} if (!$dbh); + + my $sth = $dbh->do("ALTER SESSION SET NLS_NUMERIC_CHARACTERS = '.,'") or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); +} + +sub _ora_initial_command +{ + my ($self, $dbh) = @_; + + return if ($#{ $self->{ora_initial_command} } < 0); + + $dbh = $self->{dbh} if (!$dbh); + + + # Lookup if the user have provided some sessions settings + foreach my $q (@{$self->{ora_initial_command}}) { + next if (!$q); + $self->logit("DEBUG: executing initial command to Oracle: $q\n", 1); + my $sth = $dbh->do($q) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } + +} + +sub _pg_initial_command +{ + my ($self, $dbh) = @_; + + return if ($#{ $self->{pg_initial_command} } < 0); + + $dbh = $self->{dbhdest} if (!$dbh); + + # Lookup if the user have provided some sessions settings + foreach my $q (@{$self->{pg_initial_command}}) { + $self->logit("DEBUG: executing initial command to PostgreSQL: $q\n", 1); + my $sth = $dbh->do($q) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + } + +} + + + +=head2 multiprocess_progressbar + +This function is used to display a progress bar during object scanning. + +=cut + +sub multiprocess_progressbar +{ + my ($self) = @_; + + $self->logit("Starting progressbar writer process\n", 1); + + $0 = 'ora2pg logger'; + + $| = 1; + + my $DEBUG_PBAR = 0; + my $width = 25; + my $char = '='; + my $kind = 'rows'; + my $table_count = 0; + my $table = ''; + my $global_start_time = 0; + my $total_rows = 0; + my %table_progress = (); + my $global_line_counter = 0; + + my $refresh_time = 3; #Update progress bar each 3 seconds + my $last_refresh = time(); + my $refresh_rows = 0; + + # Terminate the process when we doesn't read the complete file but must exit + local $SIG{USR1} = sub + { + if ($global_line_counter) + { + my $end_time = time(); + my $dt = $end_time - $global_start_time; + $dt ||= 1; + my $rps = int($global_line_counter / $dt); + print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'rows', "on total estimated data ($dt sec., avg: $rps tuples/sec)"), "\n"; + } + exit 0; + }; + + $pipe->reader(); + while ( my $r = <$pipe> ) + { + chomp($r); + # When quit is received, then exit immediatly + last if ($r eq 'quit'); + + # Store data export start time + if ($r =~ /^GLOBAL EXPORT START TIME: (\d+)/) + { +print STDERR "GLOBAL EXPORT START TIME: $1\n" if ($DEBUG_PBAR); + $global_start_time = $1; + } + # Store total number of tuples exported + elsif ($r =~ /^GLOBAL EXPORT ROW NUMBER: (\d+)/) + { +print STDERR "GLOBAL EXPORT ROW NUMBER: $1\n" if ($DEBUG_PBAR); + $total_rows = $1; + } + # A table export is starting (can be called multiple time with -J option) + elsif ($r =~ /TABLE EXPORT IN PROGESS: (.*?), start: (\d+), rows (\d+)/) + { +print STDERR "TABLE EXPORT IN PROGESS: $1, start: $2, rows $3\n" if ($DEBUG_PBAR); + $table_progress{$1}{start} = $2 if (!exists $table_progress{$1}{start}); + $table_progress{$1}{rows} = $3 if (!exists $table_progress{$1}{rows}); + } + # A table export is ending + elsif ($r =~ /TABLE EXPORT ENDED: (.*?), end: (\d+), rows (\d+)/) + { +print STDERR "TABLE EXPORT ENDED: $1, end: $2, rows $3\n" if ($DEBUG_PBAR); + # Store timestamp at end of table export + $table_progress{$1}{end} = $2; + + # Stores total number of rows exported when we do not used chunk of data + if (!exists $table_progress{$1}{progress}) + { + $table_progress{$1}{progress} = $3; + $global_line_counter += $3; + } + + # Display table progression + my $dt = $table_progress{$1}{end} - $table_progress{$1}{start}; + my $rps = int($table_progress{$1}{progress}/ ($dt||1)); + print STDERR $self->progress_bar($table_progress{$1}{progress}, $table_progress{$1}{rows}, 25, '=', 'rows', "Table $1 ($dt sec., $rps recs/sec)"), "\n"; + # Display global export progression + my $cur_time = time(); + $dt = $cur_time - $global_start_time; + $rps = int($global_line_counter/ ($dt || 1)); + print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'total rows', "- ($dt sec., avg: $rps recs/sec), $1 in progress."), "\r"; + $last_refresh = $cur_time; + } + # A chunk of DATA_LIMIT row is exported + elsif ($r =~ /CHUNK \d+ DUMPED: (.*?), time: (\d+), rows (\d+)/) + { +print STDERR "CHUNK X DUMPED: $1, time: $2, rows $3\n" if ($DEBUG_PBAR); + $table_progress{$1}{progress} += $3; + $global_line_counter += $3; + my $cur_time = time(); + if ($cur_time >= ($last_refresh + $refresh_time)) + { + my $dt = $cur_time - $global_start_time; + my $rps = int($global_line_counter/ ($dt || 1)); + print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'total rows', "- ($dt sec., avg: $rps recs/sec), $1 in progress."), "\r"; + $last_refresh = $cur_time; + } + } + # A table export is ending + elsif ($r =~ /TABLE EXPORT ENDED: (.*?), end: (\d+), report all parts/) + { +print STDERR "TABLE EXPORT ENDED: $1, end: $2, report all parts\n" if ($DEBUG_PBAR); + # Store timestamp at end of table export + $table_progress{$1}{end} = $2; + + # Get all statistics from multiple Oracle query + for (my $i = 0; $i < $self->{oracle_copies}; $i++) { + $table_progress{$1}{start} = $table_progress{"$1-part-$i"}{start} if (!exists $table_progress{$1}{start}); + $table_progress{$1}{rows} = $table_progress{"$1-part-$i"}{rows}; + delete $table_progress{"$1-part-$i"}; + } + + # Stores total number of rows exported when we do not used chunk of data + if (!exists $table_progress{$1}{progress}) { + $table_progress{$1}{progress} = $3; + $global_line_counter += $3; + } + + # Display table progression + my $dt = $table_progress{$1}{end} - $table_progress{$1}{start}; + my $rps = int($table_progress{$1}{rows}/ ($dt||1)); + print STDERR $self->progress_bar($table_progress{$1}{rows}, $table_progress{$1}{rows}, 25, '=', 'rows', "Table $1 ($dt sec., $rps recs/sec)"), "\n"; + } + else + { + print "PROGRESS BAR ERROR (unrecognized line sent to pipe): $r\n"; + } + + } + + if ($global_line_counter) + { + my $end_time = time(); + my $dt = $end_time - $global_start_time; + $dt ||= 1; + my $rps = int($global_line_counter / $dt); + print STDERR $self->progress_bar($global_line_counter, $total_rows, 25, '=', 'rows', "on total estimated data ($dt sec., avg: $rps tuples/sec)"), "\n"; + } + + exit 0; +} + + +=head2 progress_bar + +This function is used to display a progress bar during object scanning. + +=cut + +sub progress_bar +{ + my ($self, $got, $total, $width, $char, $kind, $msg) = @_; + + $width ||= 25; + $char ||= '='; + $kind ||= 'rows'; + my $num_width = length $total; + my $ratio = 1; + if ($total > 0) { + $ratio = $got / +$total; + } + my $len = (($width - 1) * $ratio); + $len = $width - 1 if ($len >= $width); + my $str = sprintf( + "[%-${width}s] %${num_width}s/%s $kind (%.1f%%) $msg", + $char x $len . '>', + $got, $total, 100 * $ratio + ); + $len = length($str); + $self->{prgb_len} ||= $len; + if ($len < $self->{prgb_len}) { + $str .= ' ' x ($self->{prgb_len} - $len); + } + $self->{prgb_len} = $len; + + return $str; +} + +=head2 auto_set_encoding + +This function is used to find the PostgreSQL charset corresponding to the +Oracle NLS_LANG value + +=cut + +sub auto_set_encoding +{ + my $oracle_charset = shift; + + my %ENCODING = ( + "AL32UTF8" => "UTF8", + "JA16EUC" => "EUC_JP", + "JA16SJIS" => "EUC_JIS_2004", + "ZHT32EUC" => "EUC_TW", + "CL8ISO8859P5" => "ISO_8859_5", + "AR8ISO8859P6" => "ISO_8859_6", + "EL8ISO8859P7" => "ISO_8859_7", + "IW8ISO8859P8" => "ISO_8859_8", + "CL8KOI8R" => "KOI8R", + "CL8KOI8U" => "KOI8U", + "WE8ISO8859P1" => "LATIN1", + "EE8ISO8859P2" => "LATIN2", + "SE8ISO8859P3" => "LATIN3", + "NEE8ISO8859P4"=> "LATIN4", + "WE8ISO8859P9" => "LATIN5", + "NE8ISO8859P10"=> "LATIN6", + "BLT8ISO8859P13"=> "LATIN7", + "CEL8ISO8859P14"=> "LATIN8", + "WE8ISO8859P15" => "LATIN9", + "RU8PC866" => "WIN866", + "EE8MSWIN1250" => "WIN1250", + "CL8MSWIN1251" => "WIN1251", + "WE8MSWIN1252" => "WIN1252", + "EL8MSWIN1253" => "WIN1253", + "TR8MSWIN1254" => "WIN1254", + "IW8MSWIN1255" => "WIN1255", + "AR8MSWIN1256" => "WIN1256", + "BLT8MSWIN1257"=> "WIN1257" + ); + + foreach my $k (keys %ENCODING) { + return $ENCODING{$k} if (uc($oracle_charset) eq $k); + } + + return ''; +} + +# Construct a query to exclude or only include some object wanted by the user +# following the ALLOW and EXCLUDE configuration directive. The filter returned +# must be used with the bind parameters stored in the @{$self->{query_bind_params}} +# when calling the execute() function after the call of prepare(). +sub limit_to_objects +{ + my ($self, $obj_type, $column) = @_; + + # With reports we don't have object name limitation + return if ($self->{type} eq 'SHOW_REPORT'); + + my $str = ''; + $obj_type ||= $self->{type}; + $column ||= 'TABLE_NAME'; + + my @cols = split(/\|/, $column); + my @arr_type = split(/\|/, $obj_type); + my @done = (); + my $has_limitation = 0; + $self->{query_bind_params} = (); + + for (my $i = 0; $i <= $#arr_type; $i++) { + + my $colname = $cols[0]; + $colname = $cols[$i] if (($#cols >= $i) && $cols[$i]); + + # Do not double exclusion/inclusion when column name is the same + next if (grep(/^$colname$/, @done) && ! exists $self->{limited}{$arr_type[$i]}); + push(@done, $colname); + + my $have_lookahead = 0; + if ($#{$self->{limited}{$arr_type[$i]}} >= 0) { + $str .= ' AND ('; + if ($self->{db_version} =~ /Release [89]/) { + for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { + if ($self->{limited}{$arr_type[$i]}->[$j] =~ /^\!/) { + $have_lookahead = 1; + next; + } + $str .= "upper($colname) LIKE ?"; + push(@{$self->{query_bind_params}}, uc($self->{limited}{$arr_type[$i]}->[$j])); + if ($j < $#{$self->{limited}{$arr_type[$i]}}) { + $str .= " OR "; + } + } + $str =~ s/ OR $//; + } else { + for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { + if ($self->{limited}{$arr_type[$i]}->[$j] =~ /^\!/) { + $have_lookahead = 1; + next; + } + if ($self->{is_mysql}) { + $str .= "upper($colname) RLIKE ?" ; + } else { + $str .= "REGEXP_LIKE(upper($colname), ?)" ; + } + push(@{$self->{query_bind_params}}, uc("\^$self->{limited}{$arr_type[$i]}->[$j]\$")); + if ($j < $#{$self->{limited}{$arr_type[$i]}}) { + $str .= " OR "; + } + } + $str =~ s/ OR $//; + } + $str .= ')'; + $str =~ s/ AND \(\)//; + + if ($have_lookahead) { + + if ($self->{db_version} =~ /Release [89]/) { + for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { + next if ($self->{limited}{$arr_type[$i]}->[$j] !~ /^\!(.+)/); + $str .= " AND upper($colname) NOT LIKE ?"; + push(@{$self->{query_bind_params}}, uc($1)); + } + } else { + for (my $j = 0; $j <= $#{$self->{limited}{$arr_type[$i]}}; $j++) { + next if ($self->{limited}{$arr_type[$i]}->[$j] !~ /^\!(.+)/); + if ($self->{is_mysql}) { + $str .= " AND upper($colname) NOT RLIKE ?" ; + } else { + $str .= " AND NOT REGEXP_LIKE(upper($colname), ?)" ; + } + push(@{$self->{query_bind_params}}, uc("\^$1\$")); + } + } + + } + $has_limitation = 1; + + } elsif ($#{$self->{excluded}{$arr_type[$i]}} >= 0) { + + if ($self->{db_version} =~ /Release [89]/) { + $str .= ' AND ('; + for (my $j = 0; $j <= $#{$self->{excluded}{$arr_type[$i]}}; $j++) { + $str .= "upper($colname) NOT LIKE ?" ; + push(@{$self->{query_bind_params}}, uc($self->{excluded}{$arr_type[$i]}->[$j])); + if ($j < $#{$self->{excluded}{$arr_type[$i]}}) { + $str .= " AND "; + } + } + $str .= ')'; + } else { + $str .= ' AND ('; + for (my $j = 0; $j <= $#{$self->{excluded}{$arr_type[$i]}}; $j++) { + if ($self->{is_mysql}) { + $str .= "upper($colname) NOT RLIKE ?" ; + } else { + $str .= "NOT REGEXP_LIKE(upper($colname), ?)" ; + } + push(@{$self->{query_bind_params}}, uc("\^$self->{excluded}{$arr_type[$i]}->[$j]\$")); + if ($j < $#{$self->{excluded}{$arr_type[$i]}}) { + $str .= " AND "; + } + } + $str .= ')'; + } + } + + # Always exclude unwanted tables + if (!$self->{is_mysql} && !$has_limitation && ($arr_type[$i] =~ /TABLE|SEQUENCE|VIEW|TRIGGER|TYPE|SYNONYM/)) { + if ($self->{db_version} =~ /Release [89]/) { + $str .= ' AND ('; + foreach my $t (@EXCLUDED_TABLES_8I) { + $str .= " AND upper($colname) NOT LIKE ?"; + push(@{$self->{query_bind_params}}, uc($t)); + } + $str .= ')'; + } else { + $str .= ' AND ( '; + for (my $j = 0; $j <= $#EXCLUDED_TABLES; $j++) { + if ($self->{is_mysql}) { + $str .= " upper($colname) NOT RLIKE ?" ; + } else { + $str .= " NOT REGEXP_LIKE(upper($colname), ?)" ; + } + push(@{$self->{query_bind_params}}, uc("\^$EXCLUDED_TABLES[$j]\$")); + if ($j < $#EXCLUDED_TABLES){ + $str .= " AND "; + } + } + $str .= ')'; + } + } + } + + $str =~ s/ AND \( AND/ AND \(/g; + $str =~ s/ AND \(\)//g; + $str =~ s/ OR \(\)//g; + + return uc($str); +} + + +# Preload the bytea array at lib init +BEGIN +{ + build_escape_bytea(); +} + + +=head2 _lookup_check_constraint + +This function return an array of the SQL code of the check constraints of a table + +=cut + +sub _lookup_check_constraint +{ + my ($self, $table, $check_constraint, $field_name, $nonotnull) = @_; + + my @chk_constr = (); + + my $tbsaved = $table; + $table = $self->get_replaced_tbname($table); + + # Set the check constraint definition + foreach my $k (keys %{$check_constraint->{constraint}}) + { + my $chkconstraint = $check_constraint->{constraint}->{$k}{condition}; + next if (!$chkconstraint); + my $skip_create = 0; + if (exists $check_constraint->{notnull}) { + foreach my $col (@{$check_constraint->{notnull}}) { + $skip_create = 1, last if (lc($chkconstraint) eq lc("\"$col\" IS NOT NULL")); + } + } + if (!$skip_create) + { + if (exists $self->{replaced_cols}{"\L$tbsaved\E"} && $self->{replaced_cols}{"\L$tbsaved\E"}) + { + foreach my $c (keys %{$self->{replaced_cols}{"\L$tbsaved\E"}}) + { + $chkconstraint =~ s/"$c"/"$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}"/gsi; + $chkconstraint =~ s/\b$c\b/$self->{replaced_cols}{"\L$tbsaved\E"}{"\L$c\E"}/gsi; + } + } + if ($self->{plsql_pgsql}) { + $chkconstraint = Ora2Pg::PLSQL::convert_plsql_code($self, $chkconstraint); + } + next if ($nonotnull && ($chkconstraint =~ /IS NOT NULL/)); + foreach my $c (@$field_name) { + # Force lower case + my $ret = $self->quote_object_name($c); + $chkconstraint =~ s/"$c"/$ret/igs; + $chkconstraint =~ s/\b$c\b/$ret/igs; + } + $k = $self->quote_object_name($k); + my $validate = ''; + $validate = ' NOT VALID' if ($check_constraint->{constraint}->{$k}{validate} eq 'NOT VALIDATED'); + push(@chk_constr, "ALTER TABLE $table ADD CONSTRAINT $k CHECK ($chkconstraint)$validate;\n"); + } + } + + return @chk_constr; +} + +=head2 _count_check_constraint + +This function return the number of check constraints on a given table +excluding CHECK IS NOT NULL constraint. + +=cut +sub _count_check_constraint +{ + my ($self, $check_constraint) = @_; + + my $num_chk_constr = 0; + + # Set the check constraint definition + foreach my $k (keys %{$check_constraint->{constraint}}) + { + my $chkconstraint = $check_constraint->{constraint}->{$k}{condition}; + next if (!$chkconstraint); + my $skip_create = 0; + if (exists $check_constraint->{notnull}) + { + foreach my $col (@{$check_constraint->{notnull}}) + { + $skip_create = 1, last if (lc($chkconstraint) eq lc("\"$col\" IS NOT NULL")); + } + } + if (!$skip_create) + { + $num_chk_constr++; + } + } + + return $num_chk_constr; +} + + + +=head2 _lookup_package + +This function is used to look at Oracle PACKAGE code to estimate the cost +of a migration. It return an hash: function name => function code + +=cut + +sub _lookup_package +{ + my ($self, $plsql) = @_; + + my $content = ''; + my %infos = (); + if ($plsql =~ /(?:CREATE|CREATE OR REPLACE)?\s*(?:EDITIONABLE|NONEDITIONABLE)?\s*PACKAGE\s+BODY\s*([^\s]+)((?:\s*\%ORA2PG_COMMENT\d+\%)*\s*(?:AS|IS))\s*(.*)/is) + { + my $pname = $1; + my $type = $2; + $content = $3; + $pname =~ s/"//g; + $self->logit("Looking at package $pname...\n", 1); + $content =~ s/\bEND[^;]*;$//is; + my @functions = $self->_extract_functions($content); + foreach my $f (@functions) + { + next if (!$f); + my %fct_detail = $self->_lookup_function($f, $pname); + next if (!exists $fct_detail{name}); + $fct_detail{name} =~ s/^.*\.//; + $fct_detail{name} =~ s/"//g; + %{$infos{"$pname.$fct_detail{name}"}} = %fct_detail; + } + } + + return %infos; +} + +=head2 _lookup_function + +This function is used to look at Oracle FUNCTION code to extract +all parts of a fonction + +Return a hast with the details of the function + +=cut + +sub _lookup_function +{ + my ($self, $plsql, $pname) = @_; + + if ($self->{is_mysql}) { + return Ora2Pg::MySQL::_lookup_function($self, $plsql, $pname); + } + + my %fct_detail = (); + + $fct_detail{func_ret_type} = 'OPAQUE'; + + # Split data into declarative and code part + ($fct_detail{declare}, $fct_detail{code}) = split(/\bBEGIN\b/i, $plsql, 2); + + return if (!$fct_detail{code}); + + @{$fct_detail{param_types}} = (); + $fct_detail{declare} =~ s/(\b(?:FUNCTION|PROCEDURE)\s+(?:[^\s\(]+))(\s*\%ORA2PG_COMMENT\d+\%\s*)+/$2$1 /is; + if ( ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\s\(]+)\s*(\([^\)]*\))//is) || + ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\s\(]+)\s+(RETURN|IS|AS)/$4/is) ) + { + $fct_detail{before} = $1; + $fct_detail{type} = uc($2); + $fct_detail{name} = $3; + $fct_detail{args} = $4; + + $fct_detail{fct_name} = $3; + $fct_detail{fct_name} =~ s/^[^\.]+\.//; + $fct_detail{fct_name} =~ s/"//g; + + # When the function comes from a package remove global declaration + # outside comments. They have already been extracted before. + if ($pname && $fct_detail{before}) { + $self->_remove_comments(\$fct_detail{before}); + my $cmt = ''; + while ($fct_detail{before} =~ s/(\s*\%ORA2PG_COMMENT\d+\%\s*)//is) { + # only keep comment + $cmt .= $1; + } + $fct_detail{before} = $cmt; + } + + if ($fct_detail{args} =~ /\b(RETURN|IS|AS)\b/is) { + $fct_detail{args} = '()'; + } + my $clause = ''; + my $code = ''; + $fct_detail{name} =~ s/"//g; + + $fct_detail{immutable} = 1 if ($fct_detail{declare} =~ s/\bDETERMINISTIC\b//is); + $fct_detail{setof} = 1 if ($fct_detail{declare} =~ s/\bPIPELINED\b//is); + $fct_detail{declare} =~ s/\bDEFAULT/:=/igs; + if ($fct_detail{declare} =~ s/(.*?)RETURN\s+self\s+AS RESULT IS//is) { + $fct_detail{args} .= $1; + $fct_detail{hasreturn} = 1; + $fct_detail{func_ret_type} = 'OPAQUE'; + } elsif ($fct_detail{declare} =~ s/(.*?)RETURN\s+([^\s]+)//is) { + $fct_detail{args} .= $1; + $fct_detail{hasreturn} = 1; + $fct_detail{func_ret_type} = $self->_sql_type($2) || 'OPAQUE'; + } + if ($fct_detail{declare} =~ s/(.*?)(USING|AS|IS)//is) { + $fct_detail{args} .= $1 if (!$fct_detail{hasreturn}); + $clause = $2; + } + $fct_detail{args} =~ s/;.*//s; + + if ($fct_detail{declare} =~ /LANGUAGE\s+([^\s="'><\!\(\)]+)/is) { + $fct_detail{language} = $1; + if ($fct_detail{declare} =~ /LIBRARY\s+([^\s="'><\!\(\)]+)/is) { + $fct_detail{library} = $1; + } + if ($fct_detail{declare} =~ /NAME\s+"([^"]+)"/is) { + $fct_detail{library_fct} = $1; + } + } + # rewrite argument syntax + # Replace alternate syntax for default value + $fct_detail{args} =~ s/:=/DEFAULT/igs; + # NOCOPY not supported + $fct_detail{args} =~ s/\s*NOCOPY//igs; + # IN OUT should be INOUT + $fct_detail{args} =~ s/\bIN\s+OUT/INOUT/igs; + + # Replace DEFAULT EMPTY_BLOB() from function/procedure arguments by DEFAULT NULL + $fct_detail{args} =~ s/\s+DEFAULT\s+EMPTY_[CB]LOB\(\)/DEFAULT NULL/igs; + + # Now convert types + $fct_detail{args} = Ora2Pg::PLSQL::replace_sql_type($fct_detail{args}, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + $fct_detail{declare} = Ora2Pg::PLSQL::replace_sql_type($fct_detail{declare}, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + + # Sometime variable used in FOR ... IN SELECT loop is not declared + # Append its RECORD declaration in the DECLARE section. + my $tmp_code = $fct_detail{code}; + while ($tmp_code =~ s/\bFOR\s+([^\s]+)\s+IN(.*?)LOOP//is) + { + my $varname = quotemeta($1); + my $clause = $2; + if ($fct_detail{declare} !~ /\b$varname\s+/is) { + chomp($fct_detail{declare}); + # When the cursor is refereing to a statement, declare + # it as record otherwise it don't need to be replaced + if ($clause =~ /\bSELECT\b/is) { + $fct_detail{declare} .= "\n $varname RECORD;\n"; + } + } + } + + # Set parameters for AUTONOMOUS TRANSACTION + $fct_detail{args} =~ s/\s+/ /gs; + push(@{$fct_detail{at_args}}, split(/\s*,\s*/, $fct_detail{args})); + # Remove type parts to only get parameter's name + push(@{$fct_detail{param_types}}, @{$fct_detail{at_args}}); + map { s/\s(IN|OUT|INOUT)\s/ /i; } @{$fct_detail{at_args}}; + map { s/^\(//; } @{$fct_detail{at_args}}; + map { s/^\s+//; } @{$fct_detail{at_args}}; + map { s/\s.*//; } @{$fct_detail{at_args}}; + map { s/\)$//; } @{$fct_detail{at_args}}; + @{$fct_detail{at_args}} = grep(/^.+$/, @{$fct_detail{at_args}}); + # Store type used in parameter list to lookup later for custom types + map { s/^\(//; } @{$fct_detail{param_types}}; + map { s/\)$//; } @{$fct_detail{param_types}}; + map { s/\%ORA2PG_COMMENT\d+\%//gs; } @{$fct_detail{param_types}}; + map { s/^\s*[^\s]+\s+(IN|OUT|INOUT)/$1/i; s/^((?:IN|OUT|INOUT)\s+[^\s]+)\s+[^\s]*$/$1/i; s/\(.*//; s/\s*\)\s*$//; s/\s+$//; } @{$fct_detail{param_types}}; + } else { + delete $fct_detail{func_ret_type}; + delete $fct_detail{declare}; + $fct_detail{code} = $plsql; + } + + # PostgreSQL procedure do not support OUT parameter, translate them into INOUT params + if ($self->{pg_supports_procedure} && ($fct_detail{args} =~ /\bOUT\s+[^,\)]+/i)) { + $fct_detail{args} =~ s/\bOUT(\s+[^,\)]+)/INOUT$1/igs; + } + + # Mark the function as having out parameters if any + my @nout = $fct_detail{args} =~ /\bOUT\s+([^,\)]+)/igs; + my @ninout = $fct_detail{args} =~ /\bINOUT\s+([^,\)]+)/igs; + my $nbout = $#nout+1 + $#ninout+1; + $fct_detail{inout} = 1 if ($nbout > 0); + + # Mark function as having custom type in parameter list + if ($fct_detail{inout} and $nbout > 1) { + foreach my $t (@{$fct_detail{param_types}}) { + # Consider column type reference to never be a composite type this + # is clearly not right but the false positive case might be very low + next if ($t =~ /\%TYPE/i || ($t !~ s/^(OUT|INOUT)\s+//i)); + # Mark out parameter as using composite type + if (!grep(/^\Q$t\E$/i, 'int', 'bigint', 'date', values %TYPE, values %ORA2PG_SDO_GTYPE)) { + $fct_detail{inout}++; + } + } + } + + # Collect user defined function + while ($fct_detail{declare} =~ s/\b([^\s]+)\s+EXCEPTION\s*;//) { + my $e = lc($1); + if (!exists $self->{custom_exception}{$e}) { + $self->{custom_exception}{$e} = $self->{exception_id}++; + } + } + $fct_detail{declare} =~ s/PRAGMA\s+EXCEPTION_INIT[^;]*;//igs; + + # Replace call to global variables declared in this package + foreach my $n (keys %{$self->{global_variables}}) { + next if (!$n || ($pname && (uc($n) !~ /^\U$pname\E\./))); + my $tmpname = $n; + $tmpname =~ s/^$pname\.//i; + next if ($fct_detail{code} !~ /\b$tmpname\b/is); + my $i = 0; + while ($fct_detail{code} =~ s/\b$n\s*:=\s*([^;]+)\s*;/PERFORM set_config('$n', $1, false);/is) { last if ($i++ > 100); }; + $i = 0; + while ($fct_detail{code} =~ s/([^\.]+)\b$self->{global_variables}{$n}{name}\s*:=\s*([^;]+);/$1PERFORM set_config('$n', $2, false);/is) { last if ($i++ > 100); }; + $i = 0; + while ($fct_detail{code} =~ s/([^']+)\b$n\b([^']+)/$1current_setting('$n')::$self->{global_variables}{$n}{type}$2/is) { last if ($i++ > 100); }; + $i = 0; + while ($fct_detail{code} =~ s/([^\.']+)\b$self->{global_variables}{$n}{name}\b([^']+)/$1current_setting('$n')::$self->{global_variables}{$n}{type}$2/is) { last if ($i++ > 100); }; + } + + # Replace call to raise exception + foreach my $e (keys %{$self->{custom_exception}}) { + $fct_detail{code} =~ s/\bRAISE\s+$e\b/RAISE EXCEPTION '$e' USING ERRCODE = '$self->{custom_exception}{$e}'/igs; + $fct_detail{code} =~ s/(\s+WHEN\s+)$e\s+/$1SQLSTATE '$self->{custom_exception}{$e}' /igs; + } + + return %fct_detail; +} + +#### +# Return a string to set the current search path +#### +sub set_search_path +{ + my $self = shift; + my $owner = shift; + + my $local_path = ''; + if ($self->{postgis_schema}) { + $local_path = ',' . $self->quote_object_name($self->{postgis_schema}); + } + if ($self->{data_type}{BFILE} eq 'efile') { + $local_path .= ',external_file'; + } + $local_path .= ',public'; + + my $search_path = ''; + if (!$self->{schema} && $self->{export_schema} && $owner) { + $search_path = "SET search_path = " . $self->quote_object_name($owner) . "$local_path;"; + } elsif (!$owner) { + my @pathes = (); + # When PG_SCHEMA is set, always take the value as search path + if ($self->{pg_schema}) { + @pathes = split(/\s*,\s*/, $self->{pg_schema}); + } elsif ($self->{export_schema} && $self->{schema}) { + # When EXPORT_SCHEMA is enable and we are working on a specific schema + # set it as default search_path. Useful when object are not prefixed + # with their destination schema. + push(@pathes, $self->{schema}); + } + if ($#pathes >= 0) { + map { $_ = $self->quote_object_name($_); } @pathes; + $search_path = "SET search_path = " . join(',', @pathes) . "$local_path;"; + } + } + + return "$search_path\n" if ($search_path); +} + +sub _get_human_cost +{ + my ($self, $total_cost_value) = @_; + + return 0 if (!$total_cost_value); + + my $human_cost = $total_cost_value * $self->{cost_unit_value}; + if ($human_cost >= 420) { + my $tmp = $human_cost/420; + $tmp++ if ($tmp =~ s/\.\d+//); + $human_cost = "$tmp man-day(s)"; + } else { + #my $tmp = $human_cost/60; + #$tmp++ if ($tmp =~ s/\.\d+//); + #$human_cost = "$tmp man-hour(s)"; + # mimimum to 1 day, hours are not really relevant + $human_cost = "1 man-day(s)"; + } + + return $human_cost; +} + +sub difficulty_assessment +{ + my ($self, %report_info) = @_; + + # Migration that might be run automatically + # 1 = trivial: no stored functions and no triggers + # 2 = easy: no stored functions but with triggers + # 3 = simple: stored functions and/or triggers + # Migration that need code rewrite + # 4 = manual: no stored functions but with triggers or view + # 5 = difficult: with stored functions and/or triggers + my $difficulty = 1; + + my @stored_function = ( + 'FUNCTION', + 'PACKAGE BODY', + 'PROCEDURE' + ); + + foreach my $n (@stored_function) { + if (exists $report_info{'Objects'}{$n} && $report_info{'Objects'}{$n}{'number'}) { + $difficulty = 3; + last; + } + } + if ($difficulty < 3) { + $difficulty += 1 if ( exists $report_info{'Objects'}{'TRIGGER'} && $report_info{'Objects'}{'TRIGGER'}{'number'}); + } + + + if ($difficulty < 3) { + foreach my $fct (keys %{ $report_info{'full_trigger_details'} } ) { + next if (!exists $report_info{'full_trigger_details'}{$fct}{keywords}); + $difficulty = 4; + last; + } + } + if ($difficulty <= 3) { + foreach my $fct (keys %{ $report_info{'full_view_details'} } ) { + next if (!exists $report_info{'full_view_details'}{$fct}{keywords}); + $difficulty = 4; + last; + } + } + if ($difficulty >= 3) { + foreach my $fct (keys %{ $report_info{'full_function_details'} } ) { + next if (!exists $report_info{'full_function_details'}{$fct}{keywords}); + $difficulty = 5; + last; + } + } + + my $tmp = $report_info{'total_cost_value'}/84; + $tmp++ if ($tmp =~ s/\.\d+//); + + my $level = 'A'; + $level = 'B' if ($difficulty > 3); + $level = 'C' if ( ($difficulty > 3) && ($tmp > $self->{human_days_limit}) ); + + return "$level-$difficulty"; +} + +sub _show_report +{ + my ($self, %report_info) = @_; + + my @ora_object_type = ( + 'DATABASE LINK', + 'DIRECTORY', + 'FUNCTION', + 'INDEX', + 'JOB', + 'MATERIALIZED VIEW', + 'PACKAGE BODY', + 'PROCEDURE', + 'QUERY', + 'SEQUENCE', + 'SYNONYM', + 'TABLE', + 'TABLE PARTITION', + 'TABLE SUBPARTITION', + 'TRIGGER', + 'TYPE', + 'VIEW', + +# Other object type +#CLUSTER +#CONSUMER GROUP +#DESTINATION +#DIMENSION +#EDITION +#EVALUATION CONTEXT +#INDEX PARTITION +#INDEXTYPE +#JAVA CLASS +#JAVA DATA +#JAVA RESOURCE +#JAVA SOURCE +#JOB CLASS +#LIBRARY +#LOB +#LOB PARTITION +#OPERATOR +#PACKAGE +#PROGRAM +#QUEUE +#RESOURCE PLAN +#RULE +#RULE SET +#SCHEDULE +#SCHEDULER GROUP +#TYPE BODY +#UNDEFINED +#UNIFIED AUDIT POLICY +#WINDOW +#XML SCHEMA + ); + + my $difficulty = $self->difficulty_assessment(%report_info); + my $lbl_mig_type = qq{ +Migration levels: + A - Migration that might be run automatically + B - Migration with code rewrite and a human-days cost up to $self->{human_days_limit} days + C - Migration with code rewrite and a human-days cost above $self->{human_days_limit} days +Technical levels: + 1 = trivial: no stored functions and no triggers + 2 = easy: no stored functions but with triggers, no manual rewriting + 3 = simple: stored functions and/or triggers, no manual rewriting + 4 = manual: no stored functions but with triggers or views with code rewriting + 5 = difficult: stored functions and/or triggers with code rewriting +}; + # Generate report text report + if (!$self->{dump_as_html} && !$self->{dump_as_csv} && !$self->{dump_as_sheet}) + { + my $cost_header = ''; + $cost_header = "\tEstimated cost" if ($self->{estimate_cost}); + $self->logrep("-------------------------------------------------------------------------------\n"); + $self->logrep("Ora2Pg v$VERSION - Database Migration Report\n"); + $self->logrep("-------------------------------------------------------------------------------\n"); + $self->logrep("Version\t$report_info{'Version'}\n"); + $self->logrep("Schema\t$report_info{'Schema'}\n"); + $self->logrep("Size\t$report_info{'Size'}\n\n"); + $self->logrep("-------------------------------------------------------------------------------\n"); + $self->logrep("Object\tNumber\tInvalid$cost_header\tComments\tDetails\n"); + $self->logrep("-------------------------------------------------------------------------------\n"); + foreach my $typ (sort keys %{ $report_info{'Objects'} } ) { + $report_info{'Objects'}{$typ}{'detail'} =~ s/\n/\. /gs; + if ($self->{estimate_cost}) { + $self->logrep("$typ\t$report_info{'Objects'}{$typ}{'number'}\t$report_info{'Objects'}{$typ}{'invalid'}\t$report_info{'Objects'}{$typ}{'cost_value'}\t$report_info{'Objects'}{$typ}{'comment'}\t$report_info{'Objects'}{$typ}{'detail'}\n"); + } else { + $self->logrep("$typ\t$report_info{'Objects'}{$typ}{'number'}\t$report_info{'Objects'}{$typ}{'invalid'}\t$report_info{'Objects'}{$typ}{'comment'}\t$report_info{'Objects'}{$typ}{'detail'}\n"); + } + } + $self->logrep("-------------------------------------------------------------------------------\n"); + if ($self->{estimate_cost}) { + my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); + my $comment = "$report_info{'total_cost_value'} cost migration units means approximatively $human_cost. The migration unit was set to $self->{cost_unit_value} minute(s)\n"; + $self->logrep("Total\t$report_info{'total_object_number'}\t$report_info{'total_object_invalid'}\t$report_info{'total_cost_value'}\t$comment\n"); + } else { + $self->logrep("Total\t$report_info{'total_object_number'}\t$report_info{'total_object_invalid'}\n"); + } + $self->logrep("-------------------------------------------------------------------------------\n"); + if ($self->{estimate_cost}) { + $self->logrep("Migration level : $difficulty\n"); + $self->logrep("-------------------------------------------------------------------------------\n"); + $self->logrep($lbl_mig_type); + $self->logrep("-------------------------------------------------------------------------------\n"); + if (scalar keys %{ $report_info{'full_function_details'} }) { + $self->logrep("\nDetails of cost assessment per function\n"); + foreach my $fct (sort { $report_info{'full_function_details'}{$b}{count} <=> $report_info{'full_function_details'}{$a}{count} } keys %{ $report_info{'full_function_details'} } ) { + $self->logrep("Function $fct total estimated cost: $report_info{'full_function_details'}{$fct}{count}\n"); + $self->logrep($report_info{'full_function_details'}{$fct}{info}); + } + $self->logrep("-------------------------------------------------------------------------------\n"); + } + if (scalar keys %{ $report_info{'full_trigger_details'} }) { + $self->logrep("\nDetails of cost assessment per trigger\n"); + foreach my $fct (sort { $report_info{'full_trigger_details'}{$b}{count} <=> $report_info{'full_trigger_details'}{$a}{count} } keys %{ $report_info{'full_trigger_details'} } ) { + $self->logrep("Trigger $fct total estimated cost: $report_info{'full_trigger_details'}{$fct}{count}\n"); + $self->logrep($report_info{'full_trigger_details'}{$fct}{info}); + } + $self->logrep("-------------------------------------------------------------------------------\n"); + } + if (scalar keys %{ $report_info{'full_view_details'} }) { + $self->logrep("\nDetails of cost assessment per view\n"); + foreach my $fct (sort { $report_info{'full_view_details'}{$b}{count} <=> $report_info{'full_view_details'}{$a}{count} } keys %{ $report_info{'full_view_details'} } ) { + $self->logrep("View $fct total estimated cost: $report_info{'full_view_details'}{$fct}{count}\n"); + $self->logrep($report_info{'full_view_details'}{$fct}{info}); + } + $self->logrep("-------------------------------------------------------------------------------\n"); + } + } + } + elsif ($self->{dump_as_csv}) + { + $self->logrep("-------------------------------------------------------------------------------\n"); + $self->logrep("Ora2Pg v$VERSION - Database Migration Report\n"); + $self->logrep("-------------------------------------------------------------------------------\n"); + $self->logrep("Version\t$report_info{'Version'}\n"); + $self->logrep("Schema\t$report_info{'Schema'}\n"); + $self->logrep("Size\t$report_info{'Size'}\n\n"); + $self->logrep("-------------------------------------------------------------------------------\n\n"); + $self->logrep("Object;Number;Invalid;Estimated cost;Comments\n"); + foreach my $typ (sort keys %{ $report_info{'Objects'} } ) { + $report_info{'Objects'}{$typ}{'detail'} =~ s/\n/\. /gs; + $self->logrep("$typ;$report_info{'Objects'}{$typ}{'number'};$report_info{'Objects'}{$typ}{'invalid'};$report_info{'Objects'}{$typ}{'cost_value'};$report_info{'Objects'}{$typ}{'comment'}\n"); + } + my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); + $difficulty = '' if (!$self->{estimate_cost}); + $self->logrep("\n"); + $self->logrep("Total Number;Total Invalid;Total Estimated cost;Human days cost;Migration level\n"); + $self->logrep("$report_info{'total_object_number'};$report_info{'total_object_invalid'};$report_info{'total_cost_value'};$human_cost;$difficulty\n"); + } + elsif ($self->{dump_as_sheet}) + { + $difficulty = '' if (!$self->{estimate_cost}); + my @header = ('Instance', 'Version', 'Schema', 'Size', 'Cost assessment', 'Migration type'); + my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); + my @infos = ($self->{oracle_dsn}, $report_info{'Version'}, $report_info{'Schema'}, $report_info{'Size'}, $human_cost, $difficulty); + foreach my $typ (sort @ora_object_type) { + push(@header, $typ); + $report_info{'Objects'}{$typ}{'number'} ||= 0; + $report_info{'Objects'}{$typ}{'invalid'} ||= 0; + $report_info{'Objects'}{$typ}{'cost_value'} ||= 0; + push(@infos, "$report_info{'Objects'}{$typ}{'number'}/$report_info{'Objects'}{$typ}{'invalid'}/$report_info{'Objects'}{$typ}{'cost_value'}"); + } + push(@header, "Total assessment"); + push(@infos, "$report_info{total_object_number}/$report_info{total_object_invalid}/$report_info{total_cost_value}"); + if ($self->{print_header}) { + $self->logrep('"' . join('";"', @header) . '"' . "\n"); + } + $self->logrep('"' . join('";"', @infos) . '"' . "\n"); + } + else + { + my $cost_header = ''; + $cost_header = "Estimated cost" if ($self->{estimate_cost}); + my $date = localtime(time); + my $html_header = qq{ + + + Ora2Pg - Database Migration Report + + + + + + +
+ +$cost_header +}; + + $self->logrep($html_header); + foreach my $typ (sort keys %{ $report_info{'Objects'} } ) { + $report_info{'Objects'}{$typ}{'detail'} =~ s/\n/
/gs; + $report_info{'Objects'}{$typ}{'detail'} = "
See details$report_info{'Objects'}{$typ}{'detail'}
" if ($report_info{'Objects'}{$typ}{'detail'} ne ''); + if ($self->{estimate_cost}) { + $self->logrep("\n"); + } else { + $self->logrep("\n"); + } + } + if ($self->{estimate_cost}) { + my $human_cost = $self->_get_human_cost($report_info{'total_cost_value'}); + my $comment = "$report_info{'total_cost_value'} cost migration units means approximatively $human_cost. The migration unit was set to $self->{cost_unit_value} minute(s)\n"; + $self->logrep("\n"); + } else { + $self->logrep("\n"); + } + $self->logrep("
ObjectNumberInvalidCommentsDetails
$typ$report_info{'Objects'}{$typ}{'number'}$report_info{'Objects'}{$typ}{'invalid'}$report_info{'Objects'}{$typ}{'cost_value'}$report_info{'Objects'}{$typ}{'comment'}$report_info{'Objects'}{$typ}{'detail'}
$typ$report_info{'Objects'}{$typ}{'number'}$report_info{'Objects'}{$typ}{'invalid'}$report_info{'Objects'}{$typ}{'comment'}$report_info{'Objects'}{$typ}{'detail'}
Total$report_info{'total_object_number'}$report_info{'total_object_invalid'}$report_info{'total_cost_value'}$comment
Total$report_info{'total_object_number'}$report_info{'total_object_invalid'}
\n
\n"); + if ($self->{estimate_cost}) { + $self->logrep("

Migration level: $difficulty

\n"); + $lbl_mig_type = qq{ +
    +
  • Migration levels:
    • +
      +
    • A - Migration that might be run automatically
      • +
    • B - Migration with code rewrite and a human-days cost up to $self->{human_days_limit} days
      • +
    • C - Migration with code rewrite and a human-days cost above $self->{human_days_limit} days
      • +
    +
  • Technical levels:
    • +
      +
    • 1 = trivial: no stored functions and no triggers
      • +
    • 2 = easy: no stored functions but with triggers, no manual rewriting
      • +
    • 3 = simple: stored functions and/or triggers, no manual rewriting
      • +
    • 4 = manual: no stored functions but with triggers or views with code rewriting
      • +
    • 5 = difficult: stored functions and/or triggers with code rewriting
      • +
    +
+}; + $self->logrep($lbl_mig_type); + if (scalar keys %{ $report_info{'full_function_details'} }) { + $self->logrep("

Details of cost assessment per function

\n"); + $self->logrep("
Show
    \n"); + foreach my $fct (sort { $report_info{'full_function_details'}{$b}{count} <=> $report_info{'full_function_details'}{$a}{count} } keys %{ $report_info{'full_function_details'} } ) { + + $self->logrep("
  • Function $fct total estimated cost: $report_info{'full_function_details'}{$fct}{count}
    • \n"); + $self->logrep("
      \n"); + $report_info{'full_function_details'}{$fct}{info} =~ s/\t/
    • /gs; + $report_info{'full_function_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; + $self->logrep($report_info{'full_function_details'}{$fct}{info}); + $self->logrep("
    \n"); + } + $self->logrep("
\n"); + } + if (scalar keys %{ $report_info{'full_trigger_details'} }) { + $self->logrep("

Details of cost assessment per trigger

\n"); + $self->logrep("
Show
    \n"); + foreach my $fct (sort { $report_info{'full_trigger_details'}{$b}{count} <=> $report_info{'full_trigger_details'}{$a}{count} } keys %{ $report_info{'full_trigger_details'} } ) { + + $self->logrep("
  • Trigger $fct total estimated cost: $report_info{'full_trigger_details'}{$fct}{count}
    • \n"); + $self->logrep("
      \n"); + $report_info{'full_trigger_details'}{$fct}{info} =~ s/\t/
    • /gs; + $report_info{'full_trigger_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; + $self->logrep($report_info{'full_trigger_details'}{$fct}{info}); + $self->logrep("
    \n"); + } + $self->logrep("
\n"); + } + if (scalar keys %{ $report_info{'full_view_details'} }) { + $self->logrep("

Details of cost assessment per view

\n"); + $self->logrep("
Show
    \n"); + foreach my $fct (sort { $report_info{'full_view_details'}{$b}{count} <=> $report_info{'full_view_details'}{$a}{count} } keys %{ $report_info{'full_view_details'} } ) { + + $self->logrep("
  • View $fct total estimated cost: $report_info{'full_view_details'}{$fct}{count}
    • \n"); + $self->logrep("
      \n"); + $report_info{'full_view_details'}{$fct}{info} =~ s/\t/
    • /gs; + $report_info{'full_view_details'}{$fct}{info} =~ s/\n/<\/li>\n/gs; + $self->logrep($report_info{'full_view_details'}{$fct}{info}); + $self->logrep("
    \n"); + } + $self->logrep("
\n"); + } + } + my $html_footer = qq{ + + + +}; + $self->logrep($html_footer); + } +} + +sub get_kettle_xml +{ + + return < + + template + + + + Normal + 0 + / + + + + + + + + + +ID_BATCHYID_BATCHCHANNEL_IDYCHANNEL_IDTRANSNAMEYTRANSNAMESTATUSYSTATUSLINES_READYLINES_READLINES_WRITTENYLINES_WRITTENLINES_UPDATEDYLINES_UPDATEDLINES_INPUTYLINES_INPUTLINES_OUTPUTYLINES_OUTPUTLINES_REJECTEDYLINES_REJECTEDERRORSYERRORSSTARTDATEYSTARTDATEENDDATEYENDDATELOGDATEYLOGDATEDEPDATEYDEPDATEREPLAYDATEYREPLAYDATELOG_FIELDYLOG_FIELD + + +
+ + +ID_BATCHYID_BATCHSEQ_NRYSEQ_NRLOGDATEYLOGDATETRANSNAMEYTRANSNAMESTEPNAMEYSTEPNAMESTEP_COPYYSTEP_COPYLINES_READYLINES_READLINES_WRITTENYLINES_WRITTENLINES_UPDATEDYLINES_UPDATEDLINES_INPUTYLINES_INPUTLINES_OUTPUTYLINES_OUTPUTLINES_REJECTEDYLINES_REJECTEDERRORSYERRORSINPUT_BUFFER_ROWSYINPUT_BUFFER_ROWSOUTPUT_BUFFER_ROWSYOUTPUT_BUFFER_ROWS + + +
+ +ID_BATCHYID_BATCHCHANNEL_IDYCHANNEL_IDLOG_DATEYLOG_DATELOGGING_OBJECT_TYPEYLOGGING_OBJECT_TYPEOBJECT_NAMEYOBJECT_NAMEOBJECT_COPYYOBJECT_COPYREPOSITORY_DIRECTORYYREPOSITORY_DIRECTORYFILENAMEYFILENAMEOBJECT_IDYOBJECT_IDOBJECT_REVISIONYOBJECT_REVISIONPARENT_CHANNEL_IDYPARENT_CHANNEL_IDROOT_CHANNEL_IDYROOT_CHANNEL_ID + + +
+ +ID_BATCHYID_BATCHCHANNEL_IDYCHANNEL_IDLOG_DATEYLOG_DATETRANSNAMEYTRANSNAMESTEPNAMEYSTEPNAMESTEP_COPYYSTEP_COPYLINES_READYLINES_READLINES_WRITTENYLINES_WRITTENLINES_UPDATEDYLINES_UPDATEDLINES_INPUTYLINES_INPUTLINES_OUTPUTYLINES_OUTPUTLINES_REJECTEDYLINES_REJECTEDERRORSYERRORSLOG_FIELDNLOG_FIELD + + + +
+ + 0.0 + 0.0 + + __rowset__ + 10 + 10 + N + Y + 500000 + Y + + Y + 1000 + 100 + + + + + + + + + - + 2013/02/28 14:04:49.560 + - + 2013/03/01 12:35:39.999 + + + + + __oracle_db__ + __oracle_host__ + ORACLE + Native + __oracle_instance__ + __oracle_port__ + __oracle_username__ + __oracle_password__ + + + + + EXTRA_OPTION_ORACLE.defaultRowPrefetch10000 + EXTRA_OPTION_ORACLE.fetchSize1000 + FORCE_IDENTIFIERS_TO_LOWERCASEN + FORCE_IDENTIFIERS_TO_UPPERCASEN + IS_CLUSTEREDN + PORT_NUMBER__oracle_port__ + QUOTE_ALL_FIELDSN + SUPPORTS_BOOLEAN_DATA_TYPEN + USE_POOLINGN + + + + __postgres_db__ + __postgres_host__ + POSTGRESQL + Native + __postgres_database_name__ + __postgres_port__ + __postgres_username__ + __postgres_password__ + + + + + FORCE_IDENTIFIERS_TO_LOWERCASEN + FORCE_IDENTIFIERS_TO_UPPERCASEN + IS_CLUSTEREDN + PORT_NUMBER__postgres_port__ + QUOTE_ALL_FIELDSN + SUPPORTS_BOOLEAN_DATA_TYPEY + USE_POOLINGN + EXTRA_OPTION_POSTGRESQL.synchronous_commit__sync_commit_onoff__ + + + + Table inputModified Java Script ValueY Modified Java Script ValueTable outputY + + + + Table input + TableInput + + Y + __select_copies__ + + none + + + __oracle_db__ + __select_query__ + 0 + + N + N + N + + + 122 + 160 + Y + + + + + Table output + TableOutput + + Y + __insert_copies__ + + none + + + __postgres_db__ + +
__postgres_table_name__
+ __commit_size__ + __truncate__ + Y + Y + N + N + + N + Y + N + + Y + N + + + + + + 369 + 155 + Y + + + + + Modified Java Script Value + ScriptValueMod + + Y + __js_copies__ + + none + + + N + 9 + 0 + Script 1 + for (var i=0;i<getInputRowMeta().size();i++) { + var valueMeta = getInputRowMeta().getValueMeta(i); + if (valueMeta.getTypeDesc().equals("String")) { + row[i]=replace(row[i],"\\00",''); + } +} + + + 243 + 166 + Y + + + + + + + + N + +EOF + +} + +# Constants for creating kettle files from the template +sub create_kettle_output +{ + my ($self, $table, $output_dir) = @_; + + my $oracle_host = 'localhost'; + if ($self->{oracle_dsn} =~ /host=([^;]+)/) { + $oracle_host = $1; + } + my $oracle_port = 1521; + if ($self->{oracle_dsn} =~ /port=(\d+)/) { + $oracle_port = $1; + } + my $oracle_instance=''; + if ($self->{oracle_dsn} =~ /sid=([^;]+)/) { + $oracle_instance = $1; + } elsif ($self->{oracle_dsn} =~ /dbi:Oracle:([^:]+)/) { + $oracle_instance = $1; + } + if ($self->{oracle_dsn} =~ /\/\/([^:]+):(\d+)\/(.*)/) { + $oracle_host = $1; + $oracle_port = $2; + $oracle_instance = $3; + } elsif ($self->{oracle_dsn} =~ /\/\/([^\/]+)\/(.*)/) { + $oracle_host = $1; + $oracle_instance = $2; + } + + my $pg_host = 'localhost'; + if ($self->{pg_dsn} =~ /host=([^;]+)/) { + $pg_host = $1; + } + my $pg_port = 5432; + if ($self->{pg_dsn} =~ /port=(\d+)/) { + $pg_port = $1; + } + my $pg_dbname = ''; + if ($self->{pg_dsn} =~ /dbname=([^;]+)/) { + $pg_dbname = $1; + } + + my $select_query = "SELECT * FROM $table"; + if ($self->{schema}) { + $select_query = "SELECT * FROM $self->{schema}.$table"; + } + my $select_copies = $self->{oracle_copies} || 1; + if (($self->{oracle_copies} > 1) && $self->{defined_pk}{"\L$table\E"}) { + my $colpk = $self->{defined_pk}{"\L$table\E"}; + if ($self->{preserve_case}) { + $colpk = '"' . $colpk . '"'; + } + if ($self->{schema}) { + $select_query = "SELECT * FROM $self->{schema}.$table WHERE ABS(MOD($colpk,\${Internal.Step.Unique.Count}))=\${Internal.Step.Unique.Number}"; + } else { + $select_query = "SELECT * FROM $table WHERE ABS(MOD($colpk,\${Internal.Step.Unique.Count}))=\${Internal.Step.Unique.Number}"; + } + } else { + $select_copies = 1; + } + + my $insert_copies = $self->{jobs} || 4; + my $js_copies = $insert_copies; + my $rowset = $self->{data_limit} || 10000; + if (exists $self->{local_data_limit}{$table}) { + $rowset = $self->{local_data_limit}{$table}; + } + my $commit_size = 500; + my $sync_commit_onoff = 'off'; + my $truncate = 'Y'; + $truncate = 'N' if (!$self->{truncate_table}); + + my $pg_table = $table; + if ($self->{export_schema}) { + if ($self->{pg_schema}) { + $pg_table = "$self->{pg_schema}.$table"; + } elsif ($self->{schema}) { + $pg_table = "$self->{schema}.$table"; + } + } + + my $xml = &get_kettle_xml(); + $xml =~ s/__oracle_host__/$oracle_host/gs; + $xml =~ s/__oracle_instance__/$oracle_instance/gs; + $xml =~ s/__oracle_port__/$oracle_port/gs; + $xml =~ s/__oracle_username__/$self->{oracle_user}/gs; + $xml =~ s/__oracle_password__/$self->{oracle_pwd}/gs; + $xml =~ s/__postgres_host__/$pg_host/gs; + $xml =~ s/__postgres_database_name__/$pg_dbname/gs; + $xml =~ s/__postgres_port__/$pg_port/gs; + $xml =~ s/__postgres_username__/$self->{pg_user}/gs; + $xml =~ s/__postgres_password__/$self->{pg_pwd}/gs; + $xml =~ s/__select_copies__/$select_copies/gs; + $xml =~ s/__select_query__/$select_query/gs; + $xml =~ s/__insert_copies__/$insert_copies/gs; + $xml =~ s/__js_copies__/$js_copies/gs; + $xml =~ s/__truncate__/$truncate/gs; + $xml =~ s/__transformation_name__/$table/gs; + $xml =~ s/__postgres_table_name__/$pg_table/gs; + $xml =~ s/__rowset__/$rowset/gs; + $xml =~ s/__commit_size__/$commit_size/gs; + $xml =~ s/__sync_commit_onoff__/$sync_commit_onoff/gs; + + my $fh = new IO::File; + $fh->open(">$output_dir$table.ktr") or $self->logit("FATAL: can't write to $output_dir$table.ktr, $!\n", 0, 1); + $fh->print($xml); + $self->close_export_file($fh); + + return "JAVAMAXMEM=4096 ./pan.sh -file \$KETTLE_TEMPLATE_PATH/$output_dir$table.ktr -level Detailed\n"; +} + +# Normalize SQL queries by removing parameters +sub normalize_query +{ + my ($self, $orig_query) = @_; + + return if (!$orig_query); + + # Remove comments + $orig_query =~ s/\/\*(.*?)\*\///gs; + + # Set the entire query lowercase + $orig_query = lc($orig_query); + + # Remove extra space, new line and tab characters by a single space + $orig_query =~ s/\s+/ /gs; + + # Removed start of transaction + if ($orig_query !~ /^\s*begin\s*;\s*$/) { + $orig_query =~ s/^\s*begin\s*;\s*//gs + } + + # Remove string content + $orig_query =~ s/\\'//g; + $orig_query =~ s/'[^']*'/''/g; + $orig_query =~ s/''('')+/''/g; + + # Remove NULL parameters + $orig_query =~ s/=\s*NULL/=''/g; + + # Remove numbers + $orig_query =~ s/([^a-z_\$-])-?([0-9]+)/${1}0/g; + + # Remove hexadecimal numbers + $orig_query =~ s/([^a-z_\$-])0x[0-9a-f]{1,10}/${1}0x/g; + + # Remove IN values + $orig_query =~ s/in\s*\([\'0x,\s]*\)/in (...)/g; + + return $orig_query; +} + +sub _escape_lob +{ + my ($self, $col, $generic_type, $cond, $isnested) = @_; + + if ($self->{type} eq 'COPY') { + if ( ($generic_type eq 'BLOB') || ($generic_type eq 'RAW') ) { + # RAW data type is returned in hex + $col = unpack("H*",$col) if ($generic_type ne 'RAW'); + $col = "\\\\x" . $col; + } elsif (($generic_type eq 'CLOB') || $cond->{istext}) { + $col = $self->escape_copy($col, $isnested); + } + } else { + if ( ($generic_type eq 'BLOB') || ($generic_type eq 'RAW') ) { + #$col = escape_bytea($col); + # RAW data type is returned in hex + $col = unpack("H*",$col) if ($generic_type ne 'RAW'); + if (!$self->{standard_conforming_strings}) { + $col = "'$col'"; + } else { + $col = "E'$col'"; + } + $col = "decode($col, 'hex')"; + } elsif (($generic_type eq 'CLOB') || $cond->{istext}) { + $col = $self->escape_insert($col, $isnested); + } + } + + return $col; +} + +sub escape_copy +{ + my ($self, $col, $isnested) = @_; + + my $q = "'"; + $q = '"' if ($isnested); + + if ($self->{has_utf8_fct}) { + utf8::encode($col) if (!utf8::valid($col)); + } + # Escape some character for COPY output + $col =~ s/(\0|\\|\r|\n|\t)/$ESCAPE_COPY->{$1}/gs; + if (!$self->{noescape}) { + $col =~ s/\f/\\f/gs; + $col =~ s/([\1-\10\13-\14\16-\37])/sprintf("\\%03o", ord($1))/egs; + } + + return $col; +} + +sub escape_insert +{ + my ($self, $col, $isnested) = @_; + + my $q = "'"; + $q = '"' if ($isnested); + + if (!$self->{standard_conforming_strings}) { + $col =~ s/'/''/gs; # double single quote + if ($isnested) { + $col =~ s/"/\\"/gs; # escape double quote + } + $col =~ s/\\/\\\\/gs; + $col =~ s/\0//gs; + $col = "$q$col$q"; + } else { + $col =~ s/\0//gs; + $col =~ s/\\/\\\\/gs; + $col =~ s/\r/\\r/gs; + $col =~ s/\n/\\n/gs; + if ($isnested) { + $col =~ s/'/''/gs; # double single quote + $col =~ s/"/\\"/gs; # escape double quote + $col = "$q$col$q"; + } else { + $col =~ s/'/\\'/gs; # escape single quote + $col = "E'$col'"; + } + } + return $col; +} + +sub clear_global_declaration +{ + my ($self, $pname, $str, $is_pkg_body) = @_; + + # Remove comment + $str =~ s/\%ORA2PG_COMMENT\d+\%//igs; + + # Remove all function/procedure declaration from the content + if (!$is_pkg_body) { + $str =~ s/\b(PROCEDURE|FUNCTION)\s+[^;]+;//igs; + } else { + while ($str =~ s/\b(PROCEDURE|FUNCTION)\s+.*?END[^;]*;((?:(?!\bEND\b).)*\s+(?:PROCEDURE|FUNCTION)\s+)/$2/is) { + }; + $str =~ s/(PROCEDURE|FUNCTION).*END[^;]*;//is; + } + # Remove end of the package declaration + $str =~ s/\s+END[^;]*;\s*$//igs; + # Eliminate extra newline + $str =~ s/[\r\n]+/\n/isg; + + my @cursors = (); + while ($str =~ s/(CURSOR\s+[^;]+\s+RETURN\s+[^;]+;)//is) { + push(@cursors, $1); + } + # Extract TYPE/SUBTYPE declaration + my $i = 0; + while ($str =~ s/\b(SUBTYPE|TYPE)\s+([^\s\(\)]+)\s+(AS|IS)\s+([^;]+;)//is) { + $self->{pkg_type}{$pname}{$2} = "$pname.$2"; + my $code = "$1 $self->{pkg_type}{$pname}{$2} AS $4"; + push(@{$self->{types}}, { ('name' => $2, 'code' => $code, 'pos' => $i++) }); + } + + return ($str, @cursors); +} + + +sub register_global_variable +{ + my ($self, $pname, $glob_vars) = @_; + + $glob_vars = Ora2Pg::PLSQL::replace_sql_type($glob_vars, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{$self->{data_type}}); + + # Replace PL/SQL code into PL/PGSQL similar code + $glob_vars = Ora2Pg::PLSQL::convert_plsql_code($self, $glob_vars); + + my @vars = split(/\s*(\%ORA2PG_COMMENT\d+\%|;)\s*/, $glob_vars); + map { s/^\s+//; s/\s+$//; } @vars; + my $ret = ''; + foreach my $l (@vars) + { + if ($l eq ';' || $l =~ /ORA2PG_COMMENT/ || $l =~ /^CREATE\s+/i) { + $ret .= $l if ($l ne ';'); + next; + } + next if (!$l); + $l =~ s/\-\-[^\r\n]+//sg; + $l =~ s/\s*:=\s*/ := /igs; + my ($n, $type, @others) = split(/\s+/, $l); + $ret .= $l, next if (!$type); + if (!$n) { + $n = $type; + $type = $others[0] || ''; + } + if (uc($type) eq 'EXCEPTION') { + $n = lc($n); + if (!exists $self->{custom_exception}{$n}) { + $self->{custom_exception}{$n} = $self->{exception_id}++; + } + next; + } + next if (!$pname); + my $v = lc($pname . '.' . $n); + $self->{global_variables}{$v}{name} = lc($n); + if (uc($type) eq 'CONSTANT') + { + $type = ''; + $self->{global_variables}{$v}{constant} = 1; + for (my $j = 0; $j < $#others; $j++) + { + $type .= $others[$j] if ($others[$j] ne ':=' and uc($others[$j]) ne 'DEFAULT'); + } + } + # extract the default value from the declaration + for (my $j = 0; $j < $#others; $j++) + { + $self->{global_variables}{$v}{default} = $others[$j+1] if ($others[$j] eq ':=' or uc($others[$j]) eq 'DEFAULT'); + } + if (exists $self->{global_variables}{$v}{default}) + { + $self->_restore_text_constant_part(\$self->{global_variables}{$v}{default}); + $self->{global_variables}{$v}{default} =~ s/^'//s; + $self->{global_variables}{$v}{default} =~ s/'$//s; + } + $self->{global_variables}{$v}{type} = $type; + + # Handle Oracle user defined error code + if ($self->{global_variables}{$v}{constant} && ($type =~ /bigint|int|numeric|double/) + && $self->{global_variables}{$v}{default} <= -20000 && $self->{global_variables}{$v}{default} >= -20999) + { + # Change the type into char(5) for SQLSTATE type + $self->{global_variables}{$v}{type} = 'char(5)'; + # Transform the value to match PostgreSQL user defined exceptions starting with 45 + $self->{global_variables}{$v}{default} =~ s/^-20/45/; + } + } + + return $ret; +} + +sub remove_newline +{ + my $str = shift; + + $str =~ s/[\n\r]+\s*/ /gs; + + return $str; +} + +sub _ask_username { + my $self = shift; + my $target = shift; + + print 'Enter ' . $target . ' username: '; + my $username = ReadLine(0); + chomp($username); + + return $username; +} + +sub _ask_password { + my $self = shift; + my $target = shift; + + print 'Enter ' . $target . ' password: '; + ReadMode(2); + my $password = ReadLine(0); + ReadMode(0); + chomp($password); + print "\n"; + + return $password; +} + +############## +# Prefix function calls with their package name when necessary +############## +sub normalize_function_call +{ + my ($self, $str) = @_; + + return if (!$self->{current_package}); + + my $p = lc($self->{current_package}); + + # foreach function declared in a package qualify its callis with the package name + foreach my $f (keys %{$self->{package_functions}{$p}}) { + # If the package is already prefixed to the function name in the hash take it from here + if (lc($self->{package_functions}{$p}{$f}{name}) ne lc($f)) { + $$str =~ s/([^\.])\b$f\s*([\(;])/$1$self->{package_functions}{$p}{$f}{name}$2/igs; + } elsif (exists $self->{package_functions}{$p}{$f}{package}) { + # otherwise use the package name from the hash and the function name from the string + $$str =~ s/([^\.])\b($f\s*[\(;])/$1$self->{package_functions}{$p}{$f}{package}\.$2/igs; + } + + # Append parenthesis to functions without parameters + $$str =~ s/\b($self->{package_functions}{$p}{$f}{package}\.$f)\b((?!\s*\())/$1()$2/igs; + } + # Fix unwanted double parenthesis + #$$str =~ s/\(\)\s*(\()/ $1/gs; + +} + +############## +# Requalify function calls +############## +sub requalify_function_call +{ + my ($self, $str) = @_; + + # Loop through package + foreach my $p (keys %{$self->{package_functions}}) { + # foreach function declared in a package qualify its callis with the package name + foreach my $f (keys %{$self->{package_functions}{$p}}) { + $$str =~ s/\b$p\.$f\s*([\(;])/$self->{package_functions}{$p}{$f}{name}$1/igs; + } + } +} + + +sub _make_WITH +{ + my ($with_oid, $table_info) = @_; + my @withs =(); + push @withs, 'OIDS' if ($with_oid); + push @withs, 'fillfactor=' . $table_info->{fillfactor} if (exists $table_info->{fillfactor}); + my $WITH=''; + if (@withs>0) { + $WITH .= 'WITH (' . join(",",@withs) . ')'; + } + return $WITH; +} + +sub min +{ + return $_[0] if ($_[0] < $_[1]); + + return $_[1]; +} + +1; + +__END__ + + +=head1 AUTHOR + +Gilles Darold + + +=head1 COPYRIGHT + +Copyright (c) 2000-2020 Gilles Darold - All rights reserved. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see < http://www.gnu.org/licenses/ >. + + +=head1 SEE ALSO + +L, L + + +=cut diff --git a/lib/Ora2Pg/GEOM.pm b/lib/Ora2Pg/GEOM.pm new file mode 100644 index 0000000000000000000000000000000000000000..3be1487d4a27e1284709dab10c4864362d38dd75 --- /dev/null +++ b/lib/Ora2Pg/GEOM.pm @@ -0,0 +1,830 @@ +package Ora2Pg::GEOM; +#------------------------------------------------------------------------------ +# Project : Oracle to PostgreSQL database schema converter +# Name : Ora2Pg/GEOM.pm +# Language : Perl +# Authors : Gilles Darold, gilles _AT_ darold _DOT_ net +# Copyright: Copyright (c) 2000-2020 : Gilles Darold - All rights reserved - +# Function : Perl module used to convert Oracle SDO_GEOMETRY into PostGis +# Usage : See documentation +#------------------------------------------------------------------------------ +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see < http://www.gnu.org/licenses/ >. +# +#------------------------------------------------------------------------------ +# +# Most of this work is inspired from the JTS Topology Suite developed by +# Vivid Solutions, Inc. (http://www.vividsolutions.com/jts/JTSHome.htm) +# JTS is an open source (under the LGPL license) Java library. +# See http://www.gnu.org/copyleft/lesser.html for the license. +# +#------------------------------------------------------------------------------ +# +# Special thanks to Dominique Legendre and The French Geological Survey - BRGM +# http://www.brgm.eu/ and Olivier Picavet from Oslandia http://www.oslandia.com/ +# who help me a lot with spatial understanding and their testing efforts. +# +#------------------------------------------------------------------------------ +use vars qw($VERSION); + +use strict; + +$VERSION = '21.1'; + +# SDO_ETYPE +# Second element of triplet in SDO_ELEM_INFO +my %SDO_ETYPE = ( + # code representing Point + 'POINT' => 1, + # code representing Line + 'LINESTRING' => 2, + # code representing Polygon + 'POLYGON' => 3, + # code representing compound line + 'COMPOUNDCURVE' => 4, + # code representing exterior counterclockwise polygon ring + 'POLYGON_EXTERIOR' => 1003, + # code representing interior clockwise polygon ring + 'POLYGON_INTERIOR' => 2003, + # code repersenting compound polygon counterclockwise polygon ring + 'COMPOUND_POLYGON_EXTERIOR' => 1005, + # code repersenting compound polygon clockwise polygon ring + 'COMPOUND_POLYGON_INTERIOR' => 2005, +); + +# SDO_GTYPE +# Type of the geometry +my %SDO_GTYPE = ( + # Point + 'POINT' => 1, + # Line or Curve + 'LINESTRING' => 2, + # Polygon + 'POLYGON' => 3, + # Geometry collection + 'GEOMETRYCOLLECTION' => 4, + # Multpoint + 'MULTIPOINT' => 5, + # Multiline or Multicurve + 'MULTILINESTRING' => 6, + # Multipolygon + 'MULTIPOLYGON' => 7 +); + +# SDO_INTERPRETATIONS +# Third element of triplet in SDO_ELEM_INFO +# applies to points - sdo_etype 1 +my %INTERPRETATION_POINT = ( + '0' => 'ORIENTED_POINT', + '1' => 'SIMPLE_POINT' + # n > 1: point cluster with n points +); + +# applies to lines - sdo_etype 2 +my %INTERPRETATION_LINE = ( + '1' => 'STRAIGHT_SEGMENTS', + '2' => 'CURVED_SEGMENTS' +); + +# applies to polygons - sdo_etypes 1003 and 2003 +my %INTERPRETATION_MULTI = ( + '1' => 'SIMPLE_POLY', + '2' => 'ARCS_POLY', + '3' => 'RECTANGLE', + '4' => 'CIRCLE' +); + +sub new +{ + my ($class, %options) = @_; + + # This create an OO perl object + my $self = {}; + bless ($self, $class); + + # Initialize this object + $self->_init(%options); + + # Return the instance + return($self); +} + +sub _init +{ + my ($self, %opt) = @_; + + $self->{dimension} = $opt{dimension} || -1; + $self->{srid} = $opt{srid} || -1; + $self->{geometry} = undef; + +} + +sub parse_sdo_geometry +{ + my ($self, $sdo_geom) = @_; + + # SDO_GEOMETRY DEFINITION + # CREATE TYPE sdo_geometry AS OBJECT ( + # SDO_GTYPE NUMBER, + # SDO_SRID NUMBER, + # SDO_POINT SDO_POINT_TYPE, + # SDO_ELEM_INFO SDO_ELEM_INFO_ARRAY, + # SDO_ORDINATES SDO_ORDINATE_ARRAY + # ); + #CREATE TYPE sdo_point_type AS OBJECT ( + # X NUMBER, + # Y NUMBER, + # Z NUMBER); + #CREATE TYPE sdo_elem_info_array AS VARRAY (1048576) of NUMBER; + #CREATE TYPE sdo_ordinate_array AS VARRAY (1048576) of NUMBER; + # + # SDO_ELEM_INFO + # Each triplet set of numbers is interpreted as follows: + # SDO_STARTING_OFFSET -- Indicates the offset within the SDO_ORDINATES array where the first + # ordinate for this element is stored. Offset values start at 1 and not at 0. + # SDO_ETYPE -- Indicates the type of the element. + # SDO_interpretation -- Means one of two things, depending on whether or not SDO_ETYPE is a compound element. + # If SDO_ETYPE is a compound element (4, 1005, or 2005), this field specifies how many subsequent + # triplet values are part of the element. + # If the SDO_ETYPE is not a compound element (1, 2, 1003, or 2003), the interpretation attribute determines how + # the sequence of ordinates for this element is interpreted. + + return undef if ($#{$sdo_geom} < 0); + + # Get dimension and geometry type + if ($sdo_geom->[0] =~ /^(\d)(\d)(\d{2})$/) { + $self->{geometry}{sdo_gtype} = $sdo_geom->[0] || 0; + # Extract the geometry dimension this is represented as the leftmost digit + $self->{geometry}{dim} = $1; + # Extract the linear referencing system + $self->{geometry}{lrs} = $2; + # Extract the geometry template type this is represented as the rightmost two digits + $self->{geometry}{gtype} = $3; + if ($self->{geometry}{dim} < 2) { + $self->logit("ERROR: Dimension $self->{geometry}{dim} is not valid. Either specify a dimension or use Oracle Locator Version 9i or later.\n"); + return undef; + } + } else { + $self->logit("ERROR: wrong SDO_GTYPE format in SDO_GEOMETRY data\n", 0, 0); + return undef; + } + + # Set EWKT geometry dimension + $self->{geometry}{suffix} = ''; + if ($self->{geometry}{dim} == 3) { + $self->{geometry}{suffix} = 'Z'; + } elsif ($self->{geometry}{dim} == 4) { + $self->{geometry}{suffix} = 'ZM'; + } + + # Get the srid from the data otherwise it will be + # overriden by the column srid found in meta information + $self->{geometry}{srid} = $sdo_geom->[1] if (defined $sdo_geom->[1] && $sdo_geom->[1] ne ''); + $self->{geometry}{srid} = $self->{srid} if ($self->{geometry}{srid} eq ''); + + # Look at point only coordonate + @{$self->{geometry}{sdo_point}} = (); + if ($sdo_geom->[2] =~ /^ARRAY\(0x/) { + map { if (/^[-\d]+$/) { s/,/\./; s/$/\.0/; } } @{$sdo_geom->[2]}; + push(@{$self->{geometry}{sdo_point}}, @{$sdo_geom->[2]}); + } + + # Extract elements info by triplet + @{$self->{geometry}{sdo_elem_info}} = (); + if ($sdo_geom->[3] =~ /^ARRAY\(0x/) { + push(@{$self->{geometry}{sdo_elem_info}}, @{$sdo_geom->[3]}); + } + # Extract ordinates information as arrays of dimension elements + @{$self->{geometry}{sdo_ordinates}} = (); + if ($sdo_geom->[4] =~ /^ARRAY\(0x/) { + map { if (/^[-\d]+$/) { s/,/\./; s/$/\.0/; } } @{$sdo_geom->[4]}; + push(@{$self->{geometry}{sdo_ordinates}}, @{$sdo_geom->[4]}); + } + + return $self->extract_geometry(); + +} + +# Extract geometries +sub extract_geometry +{ + my ($self, %geometry) = @_; + + my @coords = (); + + # Extract coordinates following the dimension + if ( ($self->{geometry}{gtype} == 1) && ($#{$self->{geometry}{sdo_point}} >= 0) && ($#{$self->{geometry}{sdo_elem_info}} == -1) ) { + # Single Point Type Optimization + @coords = $self->coordinates(@{$self->{geometry}{sdo_point}}); + @{$self->{geometry}{sdo_elem_info}} = ( 1, $SDO_ETYPE{POINT}, 1 ); + } else { + @coords = $self->coordinates(@{$self->{geometry}{sdo_ordinates}}); + } + + # Get the geometry + if ($self->{geometry}{gtype} == $SDO_GTYPE{POINT}) { + return $self->createPoint(0, \@coords); + } + if ($self->{geometry}{gtype} == $SDO_GTYPE{LINESTRING}) { + if ($self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUNDCURVE}) { + return $self->createCompoundLine(1, \@coords, -1); + } else { + return $self->createLine(0, \@coords); + } + } + if ($self->{geometry}{gtype} == $SDO_GTYPE{POLYGON}) { + return $self->createPolygon(0, \@coords); + } + if ($self->{geometry}{gtype} == $SDO_GTYPE{MULTIPOINT}) { + return $self->createMultiPoint(0, \@coords); + } + if ($self->{geometry}{gtype} == $SDO_GTYPE{MULTILINESTRING}) { + return $self->createMultiLine(0, \@coords, -1); + } + if ($self->{geometry}{gtype} == $SDO_GTYPE{MULTIPOLYGON}) { + return $self->createMultiPolygon(0, \@coords, -1); + } + if ($self->{geometry}{gtype} == $SDO_GTYPE{GEOMETRYCOLLECTION}) { + return $self->createCollection(0, \@coords,-1); + } +} + +# Build an array of references arrays of coordinates following the dimension +sub coordinates +{ + my ($self, @ordinates) = @_; + + my @coords = (); + my @tmp = (); + + # The number of ordinates per coordinate is taken from the dimension + for (my $i = 1; $i <= $#ordinates + 1; $i++) { + push(@tmp, $ordinates[$i - 1]); + if ($i % $self->{geometry}{dim} == 0) { + push(@coords, [(@tmp)]); + @tmp = (); + } + } + + return @coords; +} + +# Accesses the starting index in the ordinate array for the current geometry +sub get_start_offset +{ + my ($self, $t_idx) = @_; + + if ((($t_idx * 3) + 0) >= ($#{$self->{geometry}{sdo_elem_info}} + 1)) { + return -1; + } + + return $self->{geometry}{sdo_elem_info}->[($t_idx * 3) + 0]; +} + +# Get the SDO_ETYPE part from the elemInfo triplet +sub eType +{ + my ($self, $t_idx) = @_; + + if ((($t_idx * 3) + 1) >= ($#{$self->{geometry}{sdo_elem_info}}+1)) { + return -1; + } + + return $self->{geometry}{sdo_elem_info}->[($t_idx * 3) + 1]; +} + + +# Get the interpretation part the elemInfo triplet +sub interpretation +{ + my ($self, $t_idx) = @_; + + if ((($t_idx * 3) + 2) >= ($#{$self->{geometry}{sdo_elem_info}}+1)) { + return -1; + } + + return $self->{geometry}{sdo_elem_info}->[($t_idx * 3) + 2]; +} + +# Create Geometry Collection as encoded by elemInfo. +sub createCollection +{ + my ($self, $elemIndex, $coords, $numGeom) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + + my $length = $#{$coords}+1 * $self->{geometry}{dim}; + + if ($sOffset > $length) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with ordinates length " . $#{$coords}+1); + } + + my $endTriplet = ($#{$self->{geometry}{sdo_elem_info}}+1) / 3 + 1; + + my @list_geom = (); + my $etype; + my $interpretation; + my $geom; + + my $cont = 1; + for (my $i = $elemIndex; $cont && $i < $endTriplet; $i++) { + + $etype = $self->eType($i); + $interpretation = $self->interpretation($i); + + # Exclude type 0 (zero) element + next if ($etype == 0); + + if ($etype == -1) { + $cont = 0; # We reach the end of the list - get out of here + next; + + } elsif ($etype == $SDO_ETYPE{POINT}) { + + if ($interpretation == 1) { + $geom = $self->createPoint($i, $coords); + } elsif ($interpretation > 1) { + $geom = $self->createMultiPoint($i, $coords); + } + + } elsif ($etype == $SDO_ETYPE{LINESTRING}) { + + $geom = $self->createLine($i, $coords); + + } elsif ( ($etype == $SDO_ETYPE{POLYGON}) || ($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) ) { + + $geom = $self->createPolygon($i, $coords); + # Skip interior rings + while ($self->eType($i+1) == $SDO_ETYPE{POLYGON_INTERIOR}) { + $i++; + } + + } elsif ($etype == $SDO_ETYPE{POLYGON_INTERIOR}) { + $self->logit("ERROR: SDO_ETYPE 2003 (Polygon Interior) no expected in a GeometryCollection " . + "(2003 is used to represent polygon holes, in a 1003 polygon exterior)"); + next; + + } else { + $self->logit("ERROR: SDO_ETYPE $etype not representable as a EWKT Geometry by Ora2Pg."); + next; + } + push(@list_geom, $geom); + } + + return "GEOMETRYCOLLECTION$self->{geometry}{suffix} (" . join(', ', @list_geom) . ')'; +} + +# Create MultiPolygon +sub createMultiPolygon +{ + my ($self, $elemIndex, $coords, $numGeom) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + + while ($etype == 0) { + $elemIndex++; + $sOffset = $self->get_start_offset($elemIndex); + $etype = $self->eType($elemIndex); + $interpretation = $self->interpretation($elemIndex); + } + + + my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; + + if (($sOffset < 1) || ($sOffset > $length)) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with ordinates length " . ($#{$coords} + 1)); + } + # For SDO_ETYPE values 1003 and 2003, the first digit indicates exterior (1) or interior (2) + if (($etype != $SDO_ETYPE{POLYGON}) && ($etype != $SDO_ETYPE{POLYGON_EXTERIOR})) { + $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected POLYGON or POLYGON_EXTERIOR"); + } + if (($interpretation != 1) && ($interpretation != 3)) { + return undef; + } + + my $endTriplet = ($numGeom != -1) ? $elemIndex + $numGeom : (($#{$self->{geometry}{sdo_elem_info}}+1) / 3) + 1; + + my @list = (); + my $cont = 1; + + for (my $i = $elemIndex; $cont && $i < $endTriplet && ($etype = $self->eType($i)) != -1; $i++) { + # Exclude type 0 (zero) element + next if ($etype == 0); + + if (($etype == $SDO_ETYPE{POLYGON}) || ($etype == $SDO_ETYPE{POLYGON_EXTERIOR})) { + my $poly = $self->createPolygon($i, $coords); + $poly =~ s/POLYGON$self->{geometry}{suffix} //; + if ($etype != $self->eType($i-1)) { + if ( ($etype = $SDO_ETYPE{POLYGON_INTERIOR}) && ($SDO_ETYPE{POLYGON_EXTERIOR} == $self->eType($i-1)) ) { + $poly =~ s/^\(//; + $list[-1] =~ s/\)$//; + } + } + push(@list, $poly); + # Skip interior rings + while ($self->eType($i+1) == $SDO_ETYPE{POLYGON_INTERIOR}) { + $i++; + } + } else { # not a Polygon - get out here + $cont = 0; + } + } + + return "MULTIPOLYGON$self->{geometry}{suffix} (" . join(', ', @list) . ')'; +} + +# Create MultiLineString +sub createMultiLine +{ + my ($self, $elemIndex, $coords, $numGeom) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + + while ($etype == 0) { + $elemIndex++; + $sOffset = $self->get_start_offset($elemIndex); + $etype = $self->eType($elemIndex); + $interpretation = $self->interpretation($elemIndex); + } + + + my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; + + if (($sOffset < 1) || ($sOffset > $length)) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with ordinates length " . ($#{$coords} + 1)); + } + if ($etype != $SDO_ETYPE{LINESTRING}) { + $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected LINESTRING"); + } + + my $endTriplet = ($numGeom != -1) ? ($elemIndex + $numGeom) : (($#{$self->{geometry}{sdo_elem_info}} + 1) / 3); + my @list = (); + my $cont = 1; + for (my $i = $elemIndex; $cont && $i < $endTriplet && ($etype = $self->eType($i)) != -1 ; $i++) { + + # Exclude type 0 (zero) element + next if ($etype == 0); + + if ($etype == $SDO_ETYPE{LINESTRING}) { + push(@list, $self->createLine($i, $coords)); + } elsif ($etype == $SDO_ETYPE{COMPOUNDCURVE}) { + push(@list, $self->createCompoundLine(1, $coords, -1)); + } else { # not a LineString - get out of here + $cont = 0; + } + } + + if ($interpretation > 1 || grep(/CIRCULARSTRING/, @list)) { + return "MULTICURVE$self->{geometry}{suffix} (" . join(', ', @list) . ')'; + } + + map { s/LINESTRING$self->{geometry}{suffix} //; } @list; + return "MULTILINESTRING$self->{geometry}{suffix} (" . join(', ', @list) . ')'; +} + +# Create MultiPoint +sub createMultiPoint +{ + my ($self, $elemIndex, $coords) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + + while ($etype == 0) { + $elemIndex++; + $sOffset = $self->get_start_offset($elemIndex); + $etype = $self->eType($elemIndex); + $interpretation = $self->interpretation($elemIndex); + } + + my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; + + if (($sOffset < 1) || ($sOffset > $length)) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with ordinates length " . ($#{$coords} + 1)); + } + if ($etype != $SDO_ETYPE{POINT}) { + $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected POINT"); + } + + my @point = (); + my $start = ($sOffset - 1) / $self->{geometry}{dim}; + if ($interpretation > 1) { + for (my $i = $start + 1; $i <= $interpretation; $i++) { + push(@point, $self->setCoordicates($coords, $i, $i)); + } + + # Oriented point are not supported by WKT + } elsif ($interpretation != 0) { + + # There is multiple single point + my $cont = 1; + for (my $i = $start + 1; $cont && ($etype = $self->eType($i - 1)) != -1; $i++) { + # Exclude type 0 (zero) element + next if ($etype == 0); + + next if ($self->interpretation($i - 1) == 0); + if ($etype == $SDO_ETYPE{POINT}) { + push(@point, $self->setCoordicates($coords, $i, $i)); + } else { + $cont = 0; + } + } + } + my $points = "MULTIPOINT$self->{geometry}{suffix} ((" . join('), (', @point) . '))'; + + return $points; +} + +# Create Polygon +sub createPolygon +{ + my ($self, $elemIndex, $coords) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + + while ($etype == 0) { + $elemIndex++; + $sOffset = $self->get_start_offset($elemIndex); + $etype = $self->eType($elemIndex); + $interpretation = $self->interpretation($elemIndex); + } + + if ( ($sOffset < 1) || ($sOffset > ($#{$coords} + 1) * $self->{geometry}{dim}) ) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with COORDINATES length " . (($#{$coords} + 1) * $self->{geometry}{dim}) ); + } + + my @rings = (); + my $exteriorRing = $self->createLinearRing($elemIndex, $coords); + push(@rings, $exteriorRing) if ($exteriorRing); + + my $cont = 1; + for (my $i = $elemIndex+1; $cont && ($etype = $self->eType($i)) != -1; $i++) { + + # Exclude type 0 (zero) element + next if ($etype == 0); + + if ($etype == $SDO_ETYPE{LINESTRING}) { + push(@rings, $self->createLinearRing($i, $coords)); + } elsif ($etype == $SDO_ETYPE{POLYGON_INTERIOR}) { + push(@rings, $self->createLinearRing($i, $coords)); + } elsif ($etype == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) { + next; + } elsif ($etype == $SDO_ETYPE{POLYGON}) { + push(@rings, $self->createLinearRing($i, $coords)); + } else { # not a LinearRing - get out of here + $cont = 0; + } + } + + my $poly = ''; + if ($interpretation > 1) { + if ($self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) { + $poly = "CURVEPOLYGON$self->{geometry}{suffix} (COMPOUNDCURVE$self->{geometry}{suffix} (" . join(', ', @rings) . '))'; + } else { + $poly = "CURVEPOLYGON$self->{geometry}{suffix} (" . join(', ', @rings) . ')'; + } + } else { + $poly = "POLYGON$self->{geometry}{suffix} (" . join(', ', @rings) . ')'; + } + + return $poly; +} + +# Create Linear Ring for polygon +sub createLinearRing +{ + my ($self, $elemIndex, $coords) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; + + while ($etype == 0) { + $elemIndex++; + $sOffset = $self->get_start_offset($elemIndex); + $etype = $self->eType($elemIndex); + $interpretation = $self->interpretation($elemIndex); + } + + + # Exclude type 0 (zero) element + return if ($etype == 0); + + if ($sOffset > $length) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with ordinates length " . ($#{$coords} + 1)); + } + + if ( ($etype == $SDO_ETYPE{COMPOUND_POLYGON_INTERIOR}) || ($etype == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) ) { + return undef; + } + + my $ring = ''; + + my $start = ($sOffset - 1) / $self->{geometry}{dim}; + if (($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) && ($interpretation == 3)) { + my $min = $coords->[$start]; + my $max = $coords->[$start+1]; + $ring = join(' ', @$min) . ', ' . $max->[0] . ' ' . $min->[1] . ', '; + $ring .= join(' ', @$max) . ', ' . $min->[0] . ' ' . $max->[1] . ', '; + $ring .= join(' ', @$min); + } elsif (($etype == $SDO_ETYPE{POLYGON_INTERIOR}) && ($interpretation == 3)) { + my $min = $coords->[$start]; + my $max = $coords->[$start+1]; + $ring = join(' ', @$min) . ', ' . $min->[0] . ' ' . $max->[1] . ', '; + $ring .= join(' ', @$max) . ', ' . $max->[0] . ' ' . $min->[1] . ', '; + $ring .= join(' ', @$min); + } else { + my $eOffset = $self->get_start_offset($elemIndex+1); # -1 for end + my $end = ($eOffset != -1) ? (($eOffset - 1) / $self->{geometry}{dim}) : ($#{$coords} + 1); + # Polygon have the last point specified exactly the same point as the first, for others + # the coordinates for a point designating the end of one arc and the start of the next + # arc are not repeated in SDO_GEOMETRY but must be repeated in WKT. + if ( ($etype != $SDO_ETYPE{POLYGON}) || ($interpretation != 1) ) { + #$end++; + } + if ($interpretation == 2) { + if ( ($etype == $SDO_ETYPE{LINESTRING}) || ($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) || ($etype == $SDO_ETYPE{POLYGON_INTERIOR}) ) { + #$end++; + } + } + if ( ($self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUND_POLYGON_INTERIOR}) || ($self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) ) { + $end++; + } + $ring = $self->setCoordicates($coords, $start+1, $end); + if ($interpretation == 4) { + # With circle we have to repeat the first coordinates + $ring .= ', ' . $self->setCoordicates($coords, $start+1, $start+1); + } + } + + if (($etype == $SDO_ETYPE{POLYGON_EXTERIOR}) && ($interpretation == 2)) { + $ring = "CIRCULARSTRING$self->{geometry}{suffix} (" . $ring . ')'; + } elsif (($etype == $SDO_ETYPE{COMPOUND_POLYGON_EXTERIOR}) && ($interpretation == 2)) { + $ring = "COMPOUNDCURVE$self->{geometry}{suffix} (" . $ring . ')'; + } elsif ( $etype == $SDO_ETYPE{LINESTRING} && ($interpretation == 2)) { + $ring = "CIRCULARSTRING$self->{geometry}{suffix} (" . $ring . ')'; + } else { + $ring = '(' . $ring . ')'; + } + + return $ring; +} + +# Create CompoundLineString +sub createCompoundLine +{ + my ($self, $elemIndex, $coords, $numGeom) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + + while ($etype == 0) { + $elemIndex++; + $sOffset = $self->get_start_offset($elemIndex); + $etype = $self->eType($elemIndex); + $interpretation = $self->interpretation($elemIndex); + } + + + my $length = ($#{$coords} + 1) * $self->{geometry}{dim}; + + if (($sOffset < 1) || ($sOffset > $length)) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with ordinates length " . ($#{$coords} + 1)); + } + if ($etype != $SDO_ETYPE{LINESTRING}) { + $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected LINESTRING"); + } + + my $endTriplet = ($numGeom != -1) ? ($elemIndex + $numGeom) : (($#{$self->{geometry}{sdo_elem_info}} + 1) / 3); + my @list = (); + my $cont = 1; + for (my $i = $elemIndex; $cont && $i < $endTriplet && ($etype = $self->eType($i)) != -1 ; $i++) { + # Exclude type 0 (zero) element + next if ($etype == 0); + + if ($etype == $SDO_ETYPE{LINESTRING}) { + push(@list, $self->createLine($i, $coords)); + } else { # not a LineString - get out of here + $cont = 0; + } + } + + return "COMPOUNDCURVE$self->{geometry}{suffix} (" . join(', ', @list) . ')'; +} + + +# Create LineString +sub createLine +{ + my ($self, $elemIndex, $coords) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + + if ($etype != $SDO_ETYPE{LINESTRING}) { + return undef; + } + + my $start = ($sOffset - 1) / $self->{geometry}{dim}; + my $eOffset = $self->get_start_offset($elemIndex + 1); # -1 for end + my $end = ($eOffset != -1) ? (($eOffset - 1) / $self->{geometry}{dim}) : ($#{$coords} + 1); + if ( $self->{geometry}{sdo_elem_info}->[1] == $SDO_ETYPE{COMPOUNDCURVE}) { + $end++; + } + + if ($interpretation != 1) { + my $line = "CIRCULARSTRING$self->{geometry}{suffix} (" . $self->setCoordicates($coords, $start+1, $end) . ')'; + return $line; + } + + my $line = "LINESTRING$self->{geometry}{suffix} (" . $self->setCoordicates($coords, $start+1, $end) . ')'; + + return $line; +} + +# Create Point +sub createPoint +{ + my ($self, $elemIndex, $coords) = @_; + + my $sOffset = $self->get_start_offset($elemIndex); + my $etype = $self->eType($elemIndex); + my $interpretation = $self->interpretation($elemIndex); + + if (($sOffset < 1) || ($sOffset > $#{$coords} + 1)) { + $self->logit("ERROR: SDO_ELEM_INFO starting offset $sOffset inconsistent with ordinates length " . ($#{$coords} + 1)); + } + if ($etype != $SDO_ETYPE{POINT}) { + $self->logit("ERROR: SDO_ETYPE $etype inconsistent with expected POINT"); + } + # Point cluster + if ($interpretation > 1) { + return $self->createMultiPoint($elemIndex, $coords); + # Oriented point should be processed by MULTIPOINT + } elsif ($interpretation == 0) { + $self->logit("ERROR: SDO_ETYPE.POINT requires interpretation >= 1"); + return undef; + } + + my $start = ($sOffset - 1) / $self->{geometry}{dim}; + my $eOffset = $self->get_start_offset($elemIndex + 1); # -1 for end + my $end = ($eOffset != -1) ? (($eOffset - 1) / $self->{geometry}{dim}) : ($#{$coords} + 1); + my $point = "POINT$self->{geometry}{suffix} (" . $self->setCoordicates($coords, $start+1, $end) . ')'; + + return $point; +} + +sub setCoordicates +{ + my ($self, $coords, $start, $end) = @_; + + my $str = ''; + + $start ||= 1; + $end = $#{$coords} + 1 if ($end <= 0); + + for (my $i = $start - 1; $i < $end && ($i <= $#{$coords}); $i++) { + my $coordinates = join(' ', @{$coords->[$i]}); + if ($coordinates =~ /\d/) { + $str .= "$coordinates, "; + } + } + $str =~ s/, $//; + + return $str; +} + +sub logit +{ + my ($self, $message, $level, $critical) = @_; + + if (defined $self->{fhlog}) { + $self->{fhlog}->print("$message\n"); + } else { + print "$message\n"; + } +} + +1; + diff --git a/lib/Ora2Pg/MySQL.pm b/lib/Ora2Pg/MySQL.pm new file mode 100644 index 0000000000000000000000000000000000000000..64ba7ccbcbc3c61f6ab8bb7ed5333d6723396551 --- /dev/null +++ b/lib/Ora2Pg/MySQL.pm @@ -0,0 +1,1982 @@ +package Ora2Pg::MySQL; + +use vars qw($VERSION); +use strict; + +use POSIX qw(locale_h); + +#set locale to LC_NUMERIC C +setlocale(LC_NUMERIC,"C"); + + +$VERSION = '21.1'; + +# Some function might be excluded from export and assessment. +our @EXCLUDED_FUNCTION = ('SQUIRREL_GET_ERROR_OFFSET'); + +# These definitions can be overriden from configuration file +our %MYSQL_TYPE = ( + 'TINYINT' => 'smallint', # 1 byte + 'SMALLINT' => 'smallint', # 2 bytes + 'MEDIUMINT' => 'integer', # 3 bytes + 'INT' => 'integer', # 4 bytes + 'BIGINT' => 'bigint', # 8 bytes + 'DECIMAL' => 'decimal', + 'DEC' => 'decimal', + 'NUMERIC' => 'numeric', + 'FIXED' => 'numeric', + 'FLOAT' => 'double precision', + 'REAL' => 'real', + 'DOUBLE PRECISION' => 'double precision', + 'DOUBLE' => 'double precision', + 'BOOLEAN' => 'boolean', + 'BOOL' => 'boolean', + 'CHAR' => 'char', + 'VARCHAR' => 'varchar', + 'TINYTEXT' => 'text', + 'TEXT' => 'text', + 'MEDIUMTEXT' => 'text', + 'LONGTEXT' => 'text', + 'VARBINARY' => 'bytea', + 'BINARY' => 'bytea', + 'TINYBLOB' => 'bytea', + 'BLOB' => 'bytea', + 'MEDIUMBLOB' => 'bytea', + 'LONGBLOB' => 'bytea', + 'ENUM' => 'text', + 'SET' => 'text', + 'DATE' => 'date', + 'DATETIME' => 'timestamp without time zone', + 'TIME' => 'time without time zone', + 'TIMESTAMP' => 'timestamp without time zone', + 'YEAR' => 'smallint', + 'MULTIPOLYGON' => 'geometry', + 'BIT' => 'bit varying', + 'UNSIGNED' => 'bigint' +); + +sub _get_version +{ + my $self = shift; + + my $oraver = ''; + my $sql = "SELECT version()"; + + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + $oraver = $row[0]; + last; + } + $sth->finish(); + + $oraver =~ s/ \- .*//; + + return $oraver; +} + +sub _schema_list +{ + my $self = shift; + + my $sql = "SHOW DATABASES WHERE `Database` NOT IN ('information_schema', 'performance_schema');"; + + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + $sth; +} + +sub _table_exists +{ + my ($self, $schema, $table) = @_; + + my $ret = ''; + + my $sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$schema' AND TABLE_NAME = '$table'"; + + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + $ret = $row[0]; + } + $sth->finish(); + + return $ret; +} + + + +=head2 _get_encoding + +This function retrieves the Oracle database encoding + +Returns a handle to a DB query statement. + +=cut + +sub _get_encoding +{ + my ($self, $dbh) = @_; + + my $sql = "SHOW VARIABLES LIKE 'character\\_set\\_%';"; + my $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + my $my_encoding = ''; + my $my_client_encoding = ''; + while ( my @row = $sth->fetchrow()) { + if ($row[0] eq 'character_set_database') { + $my_encoding = $row[1]; + } elsif ($row[0] eq 'character_set_client') { + $my_client_encoding = $row[1]; + } + } + $sth->finish(); + + my $my_timestamp_format = ''; + my $my_date_format = ''; + $sql = "SHOW VARIABLES LIKE '%\\_format';"; + $sth = $dbh->prepare($sql) or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + $sth->execute() or $self->logit("FATAL: " . $dbh->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + if ($row[0] eq 'datetime_format') { + $my_timestamp_format = $row[1]; + } elsif ($row[0] eq 'date_format') { + $my_date_format = $row[1]; + } + } + $sth->finish(); + + #my $pg_encoding = auto_set_encoding($charset); + my $pg_encoding = $my_encoding; + + return ($my_encoding, $my_client_encoding, $pg_encoding, $my_timestamp_format, $my_date_format); +} + + + +=head2 _table_info + +This function retrieves all MySQL tables information. + +Returns a handle to a DB query statement. + +=cut + +sub _table_info +{ + my $self = shift; + + # First register all tablespace/table in memory from this database + my %tbspname = (); + my $sth = $self->{dbh}->prepare("SELECT DISTINCT TABLE_NAME, TABLESPACE_NAME FROM INFORMATION_SCHEMA.FILES WHERE table_schema = '$self->{schema}' AND TABLE_NAME IS NOT NULL") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $r = $sth->fetch) { + $tbspname{$r->[0]} = $r->[1]; + } + $sth->finish(); + + # Table: information_schema.tables + # TABLE_CATALOG | varchar(512) | NO | | | | + # TABLE_SCHEMA | varchar(64) | NO | | | | + # TABLE_NAME | varchar(64) | NO | | | | + # TABLE_TYPE | varchar(64) | NO | | | | + # ENGINE | varchar(64) | YES | | NULL | | + # VERSION | bigint(21) unsigned | YES | | NULL | | + # ROW_FORMAT | varchar(10) | YES | | NULL | | + # TABLE_ROWS | bigint(21) unsigned | YES | | NULL | | + # AVG_ROW_LENGTH | bigint(21) unsigned | YES | | NULL | | + # DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | + # MAX_DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | + # INDEX_LENGTH | bigint(21) unsigned | YES | | NULL | | + # DATA_FREE | bigint(21) unsigned | YES | | NULL | | + # AUTO_INCREMENT | bigint(21) unsigned | YES | | NULL | | + # CREATE_TIME | datetime | YES | | NULL | | + # UPDATE_TIME | datetime | YES | | NULL | | + # CHECK_TIME | datetime | YES | | NULL | | + # TABLE_COLLATION | varchar(32) | YES | | NULL | | + # CHECKSUM | bigint(21) unsigned | YES | | NULL | | + # CREATE_OPTIONS | varchar(255) | YES | | NULL | | + # TABLE_COMMENT | varchar(2048) | NO | | | | + + my %tables_infos = (); + my %comments = (); + my $sql = "SELECT TABLE_NAME,TABLE_COMMENT,TABLE_TYPE,TABLE_ROWS,ROUND( ( data_length + index_length) / 1024 / 1024, 2 ) AS \"Total Size Mb\", AUTO_INCREMENT, ENGINE FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$self->{schema}'"; + $sql .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); + $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + $row->[2] =~ s/^BASE //; + $comments{$row->[0]}{comment} = $row->[1]; + $comments{$row->[0]}{table_type} = $row->[2]; + $tables_infos{$row->[0]}{owner} = ''; + $tables_infos{$row->[0]}{num_rows} = $row->[3] || 0; + $tables_infos{$row->[0]}{comment} = $comments{$row->[0]}{comment} || ''; + $tables_infos{$row->[0]}{type} = $comments{$row->[0]}{table_type} || ''; + $tables_infos{$row->[0]}{nested} = ''; + $tables_infos{$row->[0]}{size} = $row->[4] || 0; + $tables_infos{$row->[0]}{tablespace} = 0; + $tables_infos{$row->[0]}{auto_increment} = $row->[5] || 0; + $tables_infos{$row->[0]}{tablespace} = $tbspname{$row->[0]} || ''; + + # Get creation option unavailable in information_schema + if ($row->[6] eq 'FEDERATED') { + my $sth2 = $self->{dbh}->prepare("SHOW CREATE TABLE $row->[0]") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0); + $sth2->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $r = $sth2->fetch) { + if ($r->[1] =~ /CONNECTION='([^']+)'/) { + $tables_infos{$row->[0]}{connection} = $1; + } + last; + } + $sth2->finish(); + } + } + $sth->finish(); + + return %tables_infos; +} + +sub _column_comments +{ + my ($self, $table) = @_; + + my $condition = ''; + + my $sql = "SELECT COLUMN_NAME,COLUMN_COMMENT,TABLE_NAME,'' AS \"Owner\" FROM INFORMATION_SCHEMA.COLUMNS"; + if ($self->{schema}) { + $sql .= " WHERE TABLE_SCHEMA='$self->{schema}' "; + } + $sql .= "AND TABLE_NAME='$table' " if ($table); + if (!$table) { + $sql .= $self->limit_to_objects('TABLE','TABLE_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + + my $sth = $self->{dbh}->prepare($sql) or $self->logit("WARNING only: " . $self->{dbh}->errstr . "\n", 0, 0); + + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my %data = (); + while (my $row = $sth->fetch) { + $data{$row->[2]}{$row->[0]} = $row->[1]; + } + return %data; +} + +sub _column_info +{ + my ($self, $table, $owner, $objtype, $recurs) = @_; + + $objtype ||= 'TABLE'; + + my $condition = ''; + if ($self->{schema}) { + $condition .= "AND TABLE_SCHEMA='$self->{schema}' "; + } + $condition .= "AND TABLE_NAME='$table' " if ($table); + if (!$table) { + $condition .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + $condition =~ s/^AND/WHERE/; + + # TABLE_CATALOG | varchar(512) | NO | | | | + # TABLE_SCHEMA | varchar(64) | NO | | | | + # TABLE_NAME | varchar(64) | NO | | | | + # COLUMN_NAME | varchar(64) | NO | | | | + # ORDINAL_POSITION | bigint(21) unsigned | NO | | 0 | | + # COLUMN_DEFAULT | longtext | YES | | NULL | | + # IS_NULLABLE | varchar(3) | NO | | | | + # DATA_TYPE | varchar(64) | NO | | | | + # CHARACTER_MAXIMUM_LENGTH | bigint(21) unsigned | YES | | NULL | | + # CHARACTER_OCTET_LENGTH | bigint(21) unsigned | YES | | NULL | | + # NUMERIC_PRECISION | bigint(21) unsigned | YES | | NULL | | + # NUMERIC_SCALE | bigint(21) unsigned | YES | | NULL | | + # CHARACTER_SET_NAME | varchar(32) | YES | | NULL | | + # COLLATION_NAME | varchar(32) | YES | | NULL | | + # COLUMN_TYPE | longtext | NO | | NULL | | + # COLUMN_KEY | varchar(3) | NO | | | | + # EXTRA | varchar(27) | NO | | | | + # PRIVILEGES | varchar(80) | NO | | | | + # COLUMN_COMMENT | varchar(1024) | NO | | | | + + my $str = qq{SELECT COLUMN_NAME, DATA_TYPE, CHARACTER_MAXIMUM_LENGTH, IS_NULLABLE, COLUMN_DEFAULT, NUMERIC_PRECISION, NUMERIC_SCALE, CHARACTER_OCTET_LENGTH, TABLE_NAME, '' AS OWNER, '' AS VIRTUAL_COLUMN, ORDINAL_POSITION, EXTRA, COLUMN_TYPE +FROM INFORMATION_SCHEMA.COLUMNS +$condition +ORDER BY ORDINAL_POSITION}; + # Version below 5.5 do not have DATA_TYPE column it is named DTD_IDENTIFIER + if ($self->{db_version} lt '5.5.0') { + $str =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; + } + my $sth = $self->{dbh}->prepare($str); + if (!$sth) { + $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + # Expected columns information stored in hash + # COLUMN_NAME,DATA_TYPE,DATA_LENGTH,NULLABLE,DATA_DEFAULT,DATA_PRECISION,DATA_SCALE,CHAR_LENGTH,TABLE_NAME,OWNER,VIRTUAL_COLUMN,POSITION,AUTO_INCREMENT,ENUM_INFO + my %data = (); + my $pos = 0; + while (my $row = $sth->fetch) + { + if ($row->[1] eq 'enum') { + $row->[1] = $row->[-1]; + } + $row->[10] = $pos; + push(@{$data{"$row->[8]"}{"$row->[0]"}}, @$row); + pop(@{$data{"$row->[8]"}{"$row->[0]"}}); + $pos++; + } + + return %data; +} + +sub _get_indexes +{ + my ($self, $table, $owner) = @_; + + my $condition = ''; + $condition = " FROM $self->{schema}" if ($self->{schema}); + if (!$table) { + $condition .= $self->limit_to_objects('TABLE|INDEX', "`Table`|`Key_name`"); + } else { + @{$self->{query_bind_params}} = (); + } + $condition =~ s/ AND / WHERE /; + + my %tables_infos = (); + if ($table) { + $tables_infos{$table} = 1; + } else { + %tables_infos = Ora2Pg::MySQL::_table_info($self); + } + my %data = (); + my %unique = (); + my %idx_type = (); + my %index_tablespace = (); + + # Retrieve all indexes for the given table + foreach my $t (keys %tables_infos) { + my $sth = $self->{dbh}->prepare("SHOW INDEX FROM $t $condition;") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my $i = 1; + while (my $row = $sth->fetch) { + + next if ($row->[2] eq 'PRIMARY'); + #Table : The name of the table. + #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. + #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. + #Seq_in_index : The column sequence number in the index, starting with 1. + #Column_name : The column name. + #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). + #Cardinality : An estimate of the number of unique values in the index. + #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. + #Packed : Indicates how the key is packed. NULL if it is not. + #Null : Contains YES if the column may contain NULL values and '' if not. + #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). + #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. + my $idxname = $row->[2]; + $row->[1] = 'UNIQUE' if (!$row->[1]); + $unique{$row->[0]}{$idxname} = $row->[1]; + # Set right label to spatial index + if ($row->[10] =~ /SPATIAL/) { + $row->[10] = 'SPATIAL_INDEX'; + } + $idx_type{$row->[0]}{$idxname}{type_name} = $row->[10]; + # Save original column name + my $colname = $row->[4]; + # Enclose with double quote if required + $row->[4] = $self->quote_object_name($row->[4]); + + if ($self->{preserve_case}) { + if (($row->[4] !~ /".*"/) && ($row->[4] !~ /\(.*\)/)) { + $row->[4] =~ s/^/"/; + $row->[4] =~ s/$/"/; + } + } + push(@{$data{$row->[0]}{$idxname}}, $row->[4]); + $index_tablespace{$row->[0]}{$idxname} = ''; + + } + } + + return \%unique, \%data, \%idx_type, \%index_tablespace; +} + +sub _count_indexes +{ + my ($self, $table, $owner) = @_; + + my $condition = ''; + $condition = " FROM $self->{schema}" if ($self->{schema}); + if (!$table) { + $condition .= $self->limit_to_objects('TABLE|INDEX', "`Table`|`Key_name`"); + } else { + @{$self->{query_bind_params}} = (); + } + $condition =~ s/ AND / WHERE /; + + my %tables_infos = (); + if ($table) { + $tables_infos{$table} = 1; + } else { + %tables_infos = Ora2Pg::MySQL::_table_info($self); + } + my %data = (); + + # Retrieve all indexes for the given table + foreach my $t (keys %tables_infos) { + my $sth = $self->{dbh}->prepare("SHOW INDEX FROM $t $condition;") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my $i = 1; + while (my $row = $sth->fetch) { + + #Table : The name of the table. + #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. + #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. + #Seq_in_index : The column sequence number in the index, starting with 1. + #Column_name : The column name. + #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). + #Cardinality : An estimate of the number of unique values in the index. + #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. + #Packed : Indicates how the key is packed. NULL if it is not. + #Null : Contains YES if the column may contain NULL values and '' if not. + #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). + #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. + push(@{$data{$row->[0]}{$row->[2]}}, $row->[4]); + + } + } + + return \%data; +} + + +sub _foreign_key +{ + my ($self, $table, $owner) = @_; + + my $condition = ''; + $condition .= "AND A.TABLE_NAME='$table' " if ($table); + $condition .= "AND A.CONSTRAINT_SCHEMA='$self->{schema}' " if ($self->{schema}); + + my $deferrable = $self->{fkey_deferrable} ? "'DEFERRABLE' AS DEFERRABLE" : "DEFERRABLE"; + my $sql = "SELECT DISTINCT A.COLUMN_NAME,A.ORDINAL_POSITION,A.TABLE_NAME,A.REFERENCED_TABLE_NAME,A.REFERENCED_COLUMN_NAME,A.POSITION_IN_UNIQUE_CONSTRAINT,A.CONSTRAINT_NAME,A.REFERENCED_TABLE_SCHEMA,B.MATCH_OPTION,B.UPDATE_RULE,B.DELETE_RULE FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS A INNER JOIN INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS AS B ON A.CONSTRAINT_NAME = B.CONSTRAINT_NAME WHERE A.REFERENCED_COLUMN_NAME IS NOT NULL $condition ORDER BY A.ORDINAL_POSITION,A.POSITION_IN_UNIQUE_CONSTRAINT"; + my $sth = $self->{dbh}->prepare($sql) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $sth->errstr . "\n", 0, 1); + my @cons_columns = (); + my $i = 1; + my %data = (); + my %link = (); + while (my $r = $sth->fetch) { + my $key_name = $r->[2] . '_' . $r->[0] . '_fk' . $i; + if ($r->[6] ne 'PRIMARY') { + $key_name = uc($r->[6]); + } + if ($self->{schema} && (lc($r->[7]) ne lc($self->{schema}))) { + print STDERR "WARNING: Foreign key $r->[2].$r->[0] point to an other database: $r->[7].$r->[3].$r->[4], please fix it.\n"; + } + push(@{$link{$r->[2]}{$key_name}{local}}, $r->[0]); + push(@{$link{$r->[2]}{$key_name}{remote}{$r->[3]}}, $r->[4]); + $r->[8] = 'SIMPLE'; # See pathetical documentation of mysql + # SELECT CONSTRAINT_NAME,R_CONSTRAINT_NAME,SEARCH_CONDITION,DELETE_RULE,$deferrable,DEFERRED,R_OWNER,TABLE_NAME,OWNER,UPDATE_RULE + push(@{$data{$r->[2]}}, [ ($key_name, $key_name, $r->[8], $r->[10], 'DEFERRABLE', 'Y', '', $r->[2], '', $r->[9]) ]); + $i++; + } + $sth->finish(); + + return \%link, \%data; +} + +=head2 _get_views + +This function implements an Oracle-native views information. + +Returns a hash of view names with the SQL queries they are based on. + +=cut + +sub _get_views +{ + my ($self) = @_; + + my $condition = ''; + $condition .= "AND TABLE_SCHEMA='$self->{schema}' " if ($self->{schema}); + + # Retrieve comment of each columns + # TABLE_CATALOG | varchar(512) | NO | | | | + # TABLE_SCHEMA | varchar(64) | NO | | | | + # TABLE_NAME | varchar(64) | NO | | | | + # VIEW_DEFINITION | longtext | NO | | NULL | | + # CHECK_OPTION | varchar(8) | NO | | | | + # IS_UPDATABLE | varchar(3) | NO | | | | + # DEFINER | varchar(77) | NO | | | | + # SECURITY_TYPE | varchar(7) | NO | | | | + # CHARACTER_SET_CLIENT | varchar(32) | NO | | | | + # COLLATION_CONNECTION | varchar(32) | NO | | | | + my %comments = (); + # Retrieve all views + my $str = "SELECT TABLE_NAME,VIEW_DEFINITION,CHECK_OPTION,IS_UPDATABLE,DEFINER,SECURITY_TYPE FROM INFORMATION_SCHEMA.VIEWS $condition"; + $str .= $self->limit_to_objects('VIEW', 'TABLE_NAME'); + $str .= " ORDER BY TABLE_NAME"; + $str =~ s/ AND / WHERE /; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %ordered_view = (); + my %data = (); + while (my $row = $sth->fetch) { + $row->[1] =~ s/`$self->{schema}`\.//g; + $row->[1] =~ s/`([^\s`,]+)`/$1/g; + $row->[1] =~ s/"/'/g; + $row->[1] =~ s/`/"/g; + $data{$row->[0]}{text} = $row->[1]; + $data{$row->[0]}{owner} = ''; + $data{$row->[0]}{comment} = ''; + $data{$row->[0]}{check_option} = $row->[2]; + $data{$row->[0]}{updatable} = $row->[3]; + $data{$row->[0]}{definer} = $row->[4]; + $data{$row->[0]}{security} = $row->[5]; + } + return %data; +} + +sub _get_triggers +{ + my($self) = @_; + + # Retrieve all indexes + # TRIGGER_CATALOG | varchar(512) | NO | | | | + # TRIGGER_SCHEMA | varchar(64) | NO | | | | + # TRIGGER_NAME | varchar(64) | NO | | | | + # EVENT_MANIPULATION | varchar(6) | NO | | | | + # EVENT_OBJECT_CATALOG | varchar(512) | NO | | | | + # EVENT_OBJECT_SCHEMA | varchar(64) | NO | | | | + # EVENT_OBJECT_TABLE | varchar(64) | NO | | | | + # ACTION_ORDER | bigint(4) | NO | | 0 | | + # ACTION_CONDITION | longtext | YES | | NULL | | + # ACTION_STATEMENT | longtext | NO | | NULL | | + # ACTION_ORIENTATION | varchar(9) | NO | | | | + # ACTION_TIMING | varchar(6) | NO | | | | + # ACTION_REFERENCE_OLD_TABLE | varchar(64) | YES | | NULL | | + # ACTION_REFERENCE_NEW_TABLE | varchar(64) | YES | | NULL | | + # ACTION_REFERENCE_OLD_ROW | varchar(3) | NO | | | | + # ACTION_REFERENCE_NEW_ROW | varchar(3) | NO | | | | + # CREATED | datetime | YES | | NULL | | + # SQL_MODE | varchar(8192) | NO | | | | + # DEFINER | varchar(77) | NO | | | | + # CHARACTER_SET_CLIENT | varchar(32) | NO | | | | + # COLLATION_CONNECTION | varchar(32) | NO | | | | + # DATABASE_COLLATION | varchar(32) | NO | | | | + + my $str = "SELECT TRIGGER_NAME, ACTION_TIMING, EVENT_MANIPULATION, EVENT_OBJECT_TABLE, ACTION_STATEMENT, '' AS WHEN_CLAUSE, '' AS DESCRIPTION, ACTION_ORIENTATION FROM INFORMATION_SCHEMA.TRIGGERS"; + if ($self->{schema}) { + $str .= " AND TRIGGER_SCHEMA = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','EVENT_OBJECT_TABLE|EVENT_OBJECT_TABLE|TRIGGER_NAME'); + $str =~ s/ AND / WHERE /; + + $str .= " ORDER BY EVENT_OBJECT_TABLE, TRIGGER_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my @triggers = (); + while (my $row = $sth->fetch) { + $row->[7] = 'FOR EACH '. $row->[7]; + push(@triggers, [ @$row ]); + } + + return \@triggers; +} + +sub _unique_key +{ + my($self, $table, $owner) = @_; + + my %result = (); + my @accepted_constraint_types = (); + + push @accepted_constraint_types, "'P'" unless($self->{skip_pkeys}); + push @accepted_constraint_types, "'U'" unless($self->{skip_ukeys}); + return %result unless(@accepted_constraint_types); + + # CONSTRAINT_CATALOG | varchar(512) | NO | | | | + # CONSTRAINT_SCHEMA | varchar(64) | NO | | | | + # CONSTRAINT_NAME | varchar(64) | NO | | | | + # TABLE_SCHEMA | varchar(64) | NO | | | | + # TABLE_NAME | varchar(64) | NO | | | | + # CONSTRAINT_TYPE | varchar(64) | NO | | | | + + my $condition = ''; + $condition = " FROM $self->{schema}" if ($self->{schema}); + if (!$table) { + $condition .= $self->limit_to_objects('TABLE|INDEX', "`Table`|`Key_name`"); + } else { + @{$self->{query_bind_params}} = (); + } + $condition =~ s/ AND / WHERE /; + + my %tables_infos = (); + if ($table) { + $tables_infos{$table} = 1; + } else { + %tables_infos = Ora2Pg::MySQL::_table_info($self); + } + # Retrieve all indexes for the given table + foreach my $t (keys %tables_infos) { + my $sth = $self->{dbh}->prepare("SHOW INDEX FROM $t $condition;") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my $i = 1; + while (my $row = $sth->fetch) { + # Exclude non unique constraints + next if ($row->[1]); + #Table : The name of the table. + #Non_unique : 0 if the index cannot contain duplicates, 1 if it can. + #Key_name : The name of the index. If the index is the primary key, the name is always PRIMARY. + #Seq_in_index : The column sequence number in the index, starting with 1. + #Column_name : The column name. + #Collation : How the column is sorted in the index. In MySQL, this can have values “A” (Ascending) or NULL (Not sorted). + #Cardinality : An estimate of the number of unique values in the index. + #Sub_part : The number of indexed characters if the column is only partly indexed, NULL if the entire column is indexed. + #Packed : Indicates how the key is packed. NULL if it is not. + #Null : Contains YES if the column may contain NULL values and '' if not. + #Index_type : The index method used (BTREE, FULLTEXT, HASH, RTREE). + #Comment : Information about the index not described in its own column, such as disabled if the index is disabled. + + my $idxname = $row->[0] . '_idx' . $i; + if ($row->[2] ne 'PRIMARY') { + $idxname = $row->[2]; + } + my $type = 'P'; + $type = 'U' if ($row->[2] ne 'PRIMARY'); + next if (!grep(/^'$type'$/, @accepted_constraint_types)); + my $generated = 0; + $generated = 'GENERATED NAME' if ($row->[2] ne 'PRIMARY'); + if (!exists $result{$row->[0]}{$idxname}) { + my %constraint = (type => $type, 'generated' => $generated, 'index_name' => $idxname, columns => [ ($row->[4]) ] ); + $result{$row->[0]}{$idxname} = \%constraint if ($row->[4]); + $i++ if ($row->[2] ne 'PRIMARY'); + } else { + push(@{$result{$row->[0]}{$idxname}->{columns}}, $row->[4]); + } + } + } + return %result; +} + +sub _get_functions +{ + my $self = shift; + + # Retrieve all functions + # SPECIFIC_NAME | varchar(64) | NO | | | | + # ROUTINE_CATALOG | varchar(512) | NO | | | | + # ROUTINE_SCHEMA | varchar(64) | NO | | | | + # ROUTINE_NAME | varchar(64) | NO | | | | + # ROUTINE_TYPE | varchar(9) | NO | | | | + # DATA_TYPE | varchar(64) | NO | | | | + # or DTD_IDENTIFIER < 5.5 | varchar(64) | NO | | | | + # CHARACTER_MAXIMUM_LENGTH | int(21) | YES | | NULL | | + # CHARACTER_OCTET_LENGTH | int(21) | YES | | NULL | | + # NUMERIC_PRECISION | int(21) | YES | | NULL | | + # NUMERIC_SCALE | int(21) | YES | | NULL | | + # CHARACTER_SET_NAME | varchar(64) | YES | | NULL | | + # COLLATION_NAME | varchar(64) | YES | | NULL | | + # DTD_IDENTIFIER | longtext | YES | | NULL | | + # ROUTINE_BODY | varchar(8) | NO | | | | + # ROUTINE_DEFINITION | longtext | YES | | NULL | | + # EXTERNAL_NAME | varchar(64) | YES | | NULL | | + # EXTERNAL_LANGUAGE | varchar(64) | YES | | NULL | | + # PARAMETER_STYLE | varchar(8) | NO | | | | + # IS_DETERMINISTIC | varchar(3) | NO | | | | + # SQL_DATA_ACCESS | varchar(64) | NO | | | | + # SQL_PATH | varchar(64) | YES | | NULL | | + # SECURITY_TYPE | varchar(7) | NO | | | | + # CREATED | datetime | NO | | 0000-00-00 00:00:00 | | + # LAST_ALTERED | datetime | NO | | 0000-00-00 00:00:00 | | + # SQL_MODE | varchar(8192) | NO | | | | + # ROUTINE_COMMENT | longtext | NO | | NULL | | + # DEFINER | varchar(77) | NO | | | | + # CHARACTER_SET_CLIENT | varchar(32) | NO | | | | + # COLLATION_CONNECTION | varchar(32) | NO | | | | + # DATABASE_COLLATION | varchar(32) | NO | | | | + + my $str = "SELECT ROUTINE_NAME,ROUTINE_DEFINITION,DATA_TYPE,ROUTINE_BODY,EXTERNAL_LANGUAGE,SECURITY_TYPE,IS_DETERMINISTIC FROM INFORMATION_SCHEMA.ROUTINES"; + if ($self->{schema}) { + $str .= " AND ROUTINE_SCHEMA = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('FUNCTION','ROUTINE_NAME'); + $str =~ s/ AND / WHERE /; + $str .= " ORDER BY ROUTINE_NAME"; + # Version below 5.5 do not have DATA_TYPE column it is named DTD_IDENTIFIER + if ($self->{db_version} lt '5.5.0') { + $str =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; + } + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %functions = (); + while (my $row = $sth->fetch) { + + my $kind = 'FUNCTION'; + if (!$row->[2]) { + $kind = 'PROCEDURE'; + } + next if ( ($kind ne $self->{type}) && ($self->{type} ne 'SHOW_REPORT') ); + my $sth2 = $self->{dbh}->prepare("SHOW CREATE $kind $row->[0]") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth2->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $r = $sth2->fetch) { + $functions{"$row->[0]"}{text} = $r->[2]; + last; + } + $sth2->finish(); + if ($self->{plsql_pgsql} || ($self->{type} eq 'SHOW_REPORT')) { + $functions{"$row->[0]"}{name} = $row->[0]; + $functions{"$row->[0]"}{return} = $row->[2]; + $functions{"$row->[0]"}{definition} = $row->[1]; + $functions{"$row->[0]"}{language} = $row->[3]; + $functions{"$row->[0]"}{security} = $row->[5]; + $functions{"$row->[0]"}{immutable} = $row->[6]; + } + } + + return \%functions; +} + +sub _lookup_function +{ + my ($self, $code, $fctname) = @_; + + my $type = lc($self->{type}) . 's'; + + # Replace all double quote with single quote + $code =~ s/"/'/g; + # replace backquote with double quote + $code =~ s/`/"/g; + # Remove some unused code + $code =~ s/\s+READS SQL DATA//igs; + $code =~ s/\s+UNSIGNED\b((?:.*?)\bFUNCTION\b)/$1/igs; + + my %fct_detail = (); + $fct_detail{func_ret_type} = 'OPAQUE'; + + # Split data into declarative and code part + ($fct_detail{declare}, $fct_detail{code}) = split(/\bBEGIN\b/i, $code, 2); + return if (!$fct_detail{code}); + + # Remove any label that was before the main BEGIN block + $fct_detail{declare} =~ s/\s+[^\s\:]+:\s*$//gs; + + @{$fct_detail{param_types}} = (); + + if ( ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\s\(]+)\s*(\(.*\))\s+RETURNS\s+(.*)//is) || + ($fct_detail{declare} =~ s/(.*?)\b(FUNCTION|PROCEDURE)\s+([^\s\(]+)\s*(\(.*\))//is) ) { + $fct_detail{before} = $1; + $fct_detail{type} = uc($2); + $fct_detail{name} = $3; + $fct_detail{args} = $4; + my $tmp_returned = $5; + chomp($tmp_returned); + if ($tmp_returned =~ s/\b(DECLARE\b.*)//is) { + $fct_detail{code} = $1 . $fct_detail{code}; + } + if ($fct_detail{declare} =~ s/\s*COMMENT\s+(\?TEXTVALUE\d+\?|'[^\']+')//) { + $fct_detail{comment} = $1; + } + $fct_detail{immutable} = 1 if ($fct_detail{declare} =~ s/\s*\bDETERMINISTIC\b//is); + $fct_detail{before} = ''; # There is only garbage for the moment + + $fct_detail{name} =~ s/['"]//g; + $fct_detail{fct_name} = $fct_detail{name}; + if (!$fct_detail{args}) { + $fct_detail{args} = '()'; + } + $fct_detail{immutable} = 1 if ($fct_detail{return} =~ s/\s*\bDETERMINISTIC\b//is); + $fct_detail{immutable} = 1 if ($tmp_returned =~ s/\s*\bDETERMINISTIC\b//is); + + $fctname = $fct_detail{name} || $fctname; + if ($type eq 'functions' && exists $self->{$type}{$fctname}{return} && $self->{$type}{$fctname}{return}) { + $fct_detail{hasreturn} = 1; + $fct_detail{func_ret_type} = $self->_sql_type($self->{$type}{$fctname}{return}); + } elsif ($type eq 'functions' && !exists $self->{$type}{$fctname}{return} && $tmp_returned) { + $tmp_returned =~ s/\s+CHARSET.*//is; + $fct_detail{func_ret_type} = $self->_sql_type($tmp_returned); + $fct_detail{hasreturn} = 1; + } + $fct_detail{language} = $self->{$type}{$fctname}{language}; + $fct_detail{immutable} = 1 if ($self->{$type}{$fctname}{immutable} eq 'YES'); + $fct_detail{security} = $self->{$type}{$fctname}{security}; + + # Procedure that have out parameters are functions with PG + if ($type eq 'procedures' && $fct_detail{args} =~ /\b(OUT|INOUT)\b/) { + # set return type to empty to avoid returning void later + $fct_detail{func_ret_type} = ' '; + } + # IN OUT should be INOUT + $fct_detail{args} =~ s/\bIN\s+OUT/INOUT/igs; + + # Move the DECLARE statement from code to the declare section. + $fct_detail{declare} = ''; + while ($fct_detail{code} =~ s/DECLARE\s+([^;]+;)//is) { + $fct_detail{declare} .= "\n$1"; + } + # Now convert types + if ($fct_detail{args}) { + $fct_detail{args} = replace_sql_type($fct_detail{args}, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{ $self->{data_type} }); + } + if ($fct_detail{declare}) { + $fct_detail{declare} = replace_sql_type($fct_detail{declare}, $self->{pg_numeric_type}, $self->{default_numeric}, $self->{pg_integer_type}, %{ $self->{data_type} }); + } + + $fct_detail{args} =~ s/\s+/ /gs; + push(@{$fct_detail{param_types}}, split(/\s*,\s*/, $fct_detail{args})); + # Store type used in parameter list to lookup later for custom types + map { s/^\(//; } @{$fct_detail{param_types}}; + map { s/\)$//; } @{$fct_detail{param_types}}; + map { s/\%ORA2PG_COMMENT\d+\%//gs; } @{$fct_detail{param_types}}; + map { s/^\s*[^\s]+\s+(IN|OUT|INOUT)/$1/i; s/^((?:IN|OUT|INOUT)\s+[^\s]+)\s+[^\s]*$/$1/i; s/\(.*//; s/\s*\)\s*$//; s/\s+$//; } @{$fct_detail{param_types}}; + + } else { + delete $fct_detail{func_ret_type}; + delete $fct_detail{declare}; + $fct_detail{code} = $code; + } + + # Mark the function as having out parameters if any + my @nout = $fct_detail{args} =~ /\bOUT\s+([^,\)]+)/igs; + my @ninout = $fct_detail{args} =~ /\bINOUT\s+([^,\)]+)/igs; + my $nbout = $#nout+1 + $#ninout+1; + $fct_detail{inout} = 1 if ($nbout > 0); + + ($fct_detail{code}, $fct_detail{declare}) = replace_mysql_variables($self, $fct_detail{code}, $fct_detail{declare}); + + return %fct_detail; +} + +sub replace_mysql_variables +{ + my ($self, $code, $declare) = @_; + + # Look for mysql global variables and add them to the custom variable list + while ($code =~ s/\b(?:SET\s+)?\@\@(?:SESSION\.)?([^\s:=]+)\s*:=\s*([^;]+);/PERFORM set_config('$1', $2, false);/is) { + my $n = $1; + my $v = $2; + $self->{global_variables}{$n}{name} = lc($n); + # Try to set a default type for the variable + $self->{global_variables}{$n}{type} = 'bigint'; + if ($v =~ /'[^\']*'/) { + $self->{global_variables}{$n}{type} = 'varchar'; + } + if ($n =~ /datetime/i) { + $self->{global_variables}{$n}{type} = 'timestamp'; + } elsif ($n =~ /time/i) { + $self->{global_variables}{$n}{type} = 'time'; + } elsif ($n =~ /date/i) { + $self->{global_variables}{$n}{type} = 'date'; + } + } + + my @to_be_replaced = (); + # Look for local variable definition and append them to the declare section + while ($code =~ s/SET\s+\@([^\s:]+)\s*:=\s*([^;]+);/SET $1 = $2;/is) { + my $n = $1; + my $v = $2; + # Try to set a default type for the variable + my $type = 'integer'; + $type = 'varchar' if ($v =~ /'[^']*'/); + if ($n =~ /datetime/i) { + $type = 'timestamp'; + } elsif ($n =~ /time/i) { + $type = 'time'; + } elsif ($n =~ /date/i) { + $type = 'date'; + } + $declare .= "$n $type;\n" if ($declare !~ /\b$n $type;/s); + push(@to_be_replaced, $n); + } + + # Look for local variable definition and append them to the declare section + while ($code =~ s/(\s+)\@([^\s:=]+)\s*:=\s*([^;]+);/$1$2 := $3;/is) { + my $n = $2; + my $v = $3; + # Try to set a default type for the variable + my $type = 'integer'; + $type = 'varchar' if ($v =~ /'[^']*'/); + if ($n =~ /datetime/i) { + $type = 'timestamp'; + } elsif ($n =~ /time/i) { + $type = 'time'; + } elsif ($n =~ /date/i) { + $type = 'date'; + } + $declare .= "$n $type;\n" if ($declare !~ /\b$n $type;/s); + push(@to_be_replaced, $n); + } + + # Fix other call to the same variable in the code + foreach my $n (@to_be_replaced) { + $code =~ s/\@$n\b(\s*[^:])/$n$1/gs; + } + + # Look for local variable definition and append them to the declare section + while ($code =~ s/\@([a-z0-9_]+)/$1/is) { + my $n = $1; + # Try to set a default type for the variable + my $type = 'varchar'; + if ($n =~ /datetime/i) { + $type = 'timestamp'; + } elsif ($n =~ /time/i) { + $type = 'time'; + } elsif ($n =~ /date/i) { + $type = 'date'; + } + $declare .= "$n $type;\n" if ($declare !~ /\b$n $type;/s); + # Fix other call to the same variable in the code + $code =~ s/\@$n\b/$n/gs; + } + + # Look for variable definition with SELECT statement + $code =~ s/\bSET\s+([^\s=]+)\s*=\s*([^;]+\bSELECT\b[^;]+);/$1 = $2;/igs; + + return ($code, $declare); +} + +sub _list_all_funtions +{ + my $self = shift; + + # Retrieve all functions + # ROUTINE_SCHEMA | varchar(64) | NO | | | | + # ROUTINE_NAME | varchar(64) | NO | | | | + # ROUTINE_TYPE | varchar(9) | NO | | | | + + my $str = "SELECT ROUTINE_NAME,DATA_TYPE FROM INFORMATION_SCHEMA.ROUTINES"; + if ($self->{schema}) { + $str .= " AND ROUTINE_SCHEMA = '$self->{schema}'"; + } + if ($self->{db_version} lt '5.5.0') { + $str =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; + } + $str .= " " . $self->limit_to_objects('FUNCTION','ROUTINE_NAME'); + $str =~ s/ AND / WHERE /; + $str .= " ORDER BY ROUTINE_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my @functions = (); + while (my $row = $sth->fetch) { + push(@functions, $row->[0]); + } + $sth->finish(); + + return @functions; +} + + + +sub _sql_type +{ + my ($self, $type, $len, $precision, $scale) = @_; + + my $data_type = ''; + + # Simplify timestamp type + $type =~ s/TIMESTAMP\s*\(\s*\d+\s*\)/TIMESTAMP/i; + $type =~ s/TIME\s*\(\s*\d+\s*\)/TIME/i; + $type =~ s/DATE\s*\(\s*\d+\s*\)/DATE/i; + # Remove BINARY from CHAR(n) BINARY, TEXT(n) BINARY, VARCHAR(n) BINARY ... + $type =~ s/(CHAR|TEXT)\s*(\(\s*\d+\s*\)) BINARY/$1$2/i; + $type =~ s/(CHAR|TEXT)\s+BINARY/$1/i; + + # Some length and scale may have not been extracted before + if ($type =~ s/\(\s*(\d+)\s*\)//) { + $len = $1; + } elsif ($type =~ s/\(\s*(\d+)\s*,\s*(\d+)\s*\)//) { + $len = $1; + $scale = $2; + } + if ($type !~ /CHAR/i) { + $precision = $len if (!$precision); + } + + # Override the length + $len = $precision if ( ((uc($type) eq 'NUMBER') || (uc($type) eq 'BIT')) && $precision ); + if (exists $self->{data_type}{uc($type)}) { + $type = uc($type); # Force uppercase + if ($len) { + if ( ($type eq "CHAR") || ($type =~ /VARCHAR/) ) { + # Type CHAR have default length set to 1 + # Type VARCHAR(2) must have a specified length + $len = 1 if (!$len && ($type eq "CHAR")); + return "$self->{data_type}{$type}($len)"; + } elsif ($type eq 'BIT') { + if ($precision) { + return "$self->{data_type}{$type}($precision)"; + } else { + return $self->{data_type}{$type}; + } + } elsif ($type =~ /(TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT|REAL|DOUBLE|FLOAT|DECIMAL|NUMERIC)/i) { + # This is an integer + if (!$scale) { + if ($precision) { + if ($self->{pg_integer_type}) { + if ($precision < 5) { + return 'smallint'; + } elsif ($precision <= 9) { + return 'integer'; # The speediest in PG + } else { + return 'bigint'; + } + } + return "numeric($precision)"; + } else { + # Most of the time interger should be enought? + return $self->{data_type}{$type}; + } + } else { + if ($precision) { + if ($type !~ /DOUBLE/ && $self->{pg_numeric_type}) { + if ($precision <= 6) { + return 'real'; + } else { + return 'double precision'; + } + } + return "decimal($precision,$scale)"; + } + } + } + return $self->{data_type}{$type}; + } else { + return $self->{data_type}{$type}; + } + } + + return $type; +} + +sub replace_sql_type +{ + my ($str, $pg_numeric_type, $default_numeric, $pg_integer_type, %data_type) = @_; + + $str =~ s/with local time zone/with time zone/igs; + $str =~ s/([A-Z])ORA2PG_COMMENT/$1 ORA2PG_COMMENT/igs; + + # Remove any reference to UNSIGNED AND ZEROFILL + # but translate CAST( ... AS unsigned) before. + $str =~ s/(\s+AS\s+)UNSIGNED/$1$data_type{'UNSIGNED'}/gis; + $str =~ s/\b(UNSIGNED|ZEROFILL)\b//gis; + + # Remove BINARY from CHAR(n) BINARY and VARCHAR(n) BINARY + $str =~ s/(CHAR|TEXT)\s*(\(\s*\d+\s*\))\s+BINARY/$1$2/gis; + $str =~ s/(CHAR|TEXT)\s+BINARY/$1/gis; + + # Replace type with precision + my $mysqltype_regex = ''; + foreach (keys %data_type) { + $mysqltype_regex .= quotemeta($_) . '|'; + } + $mysqltype_regex =~ s/\|$//; + while ($str =~ /(.*)\b($mysqltype_regex)\s*\(([^\)]+)\)/i) { + my $backstr = $1; + my $type = uc($2); + my $args = $3; + if (uc($type) eq 'ENUM') { + # Prevent from infinit loop + $str =~ s/\(/\%\|/s; + $str =~ s/\)/\%\|\%/s; + next; + } + if (exists $data_type{"$type($args)"}) { + $str =~ s/\b$type\($args\)/$data_type{"$type($args)"}/igs; + next; + } + if ($backstr =~ /_$/) { + $str =~ s/\b($mysqltype_regex)\s*\(([^\)]+)\)/$1\%\|$2\%\|\%/is; + next; + } + + my ($precision, $scale) = split(/,/, $args); + $scale ||= 0; + my $len = $precision || 0; + $len =~ s/\D//; + if ( $type =~ /CHAR/i ) { + # Type CHAR have default length set to 1 + # Type VARCHAR must have a specified length + $len = 1 if (!$len && ($type eq "CHAR")); + $str =~ s/\b$type\b\s*\([^\)]+\)/$data_type{$type}\%\|$len\%\|\%/is; + } elsif ($precision && ($type =~ /(BIT|TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT|REAL|DOUBLE|FLOAT|DECIMAL|NUMERIC)/)) { + if (!$scale) { + if ($type =~ /(BIT|TINYINT|SMALLINT|MEDIUMINT|INTEGER|BIGINT|INT)/) { + if ($pg_integer_type) { + if ($precision < 5) { + $str =~ s/\b$type\b\s*\([^\)]+\)/smallint/is; + } elsif ($precision <= 9) { + $str =~ s/\b$type\b\s*\([^\)]+\)/integer/is; + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/bigint/is; + } + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/numeric\%\|$precision\%\|\%/i; + } + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/$data_type{$type}\%\|$precision\%\|\%/is; + } + } else { + if ($type =~ /DOUBLE/) { + $str =~ s/\b$type\b\s*\([^\)]+\)/decimal\%\|$args\%\|\%/is; + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/$data_type{$type}\%\|$args\%\|\%/is; + } + } + } else { + # Prevent from infinit loop + $str =~ s/\(/\%\|/s; + $str =~ s/\)/\%\|\%/s; + } + } + $str =~ s/\%\|\%/\)/gs; + $str =~ s/\%\|/\(/gs; + + # Replace datatype even without precision + my %recover_type = (); + my $i = 0; + foreach my $type (sort { length($b) <=> length($a) } keys %data_type) { + # Keep enum as declared, we are not in table definition + next if (uc($type) eq 'ENUM'); + while ($str =~ s/\b$type\b/%%RECOVER_TYPE$i%%/is) { + $recover_type{$i} = $data_type{$type}; + $i++; + } + } + + foreach $i (keys %recover_type) { + $str =~ s/\%\%RECOVER_TYPE$i\%\%/$recover_type{$i}/; + } + + # Set varchar without length to text + $str =~ s/\bVARCHAR(\s*(?!\())/text$1/igs; + + return $str; +} + +sub _get_job +{ + my($self) = @_; + + # Retrieve all database job from user_jobs table + my $str = "SELECT EVENT_NAME,EVENT_DEFINITION,EXECUTE_AT FROM INFORMATION_SCHEMA.EVENTS WHERE STATUS = 'ENABLED'"; + if ($self->{schema}) { + $str .= " AND EVENT_SCHEMA = '$self->{schema}'"; + } + $str .= $self->limit_to_objects('JOB', 'EVENT_NAME'); + $str .= " ORDER BY EVENT_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + $data{$row->[0]}{what} = $row->[1]; + $data{$row->[0]}{interval} = $row->[2]; + } + + return %data; +} + +sub _get_dblink +{ + my($self) = @_; + + # Must be able to read mysql.servers table + return if ($self->{user_grants}); + + # Retrieve all database link from dba_db_links table + my $str = "SELECT OWNER,SERVER_NAME,USERNAME,HOST,DB,PORT,PASSWORD FROM mysql.servers"; + $str .= $self->limit_to_objects('DBLINK', 'SERVER_NAME'); + $str .= " ORDER BY SERVER_NAME"; + $str =~ s/mysql.servers AND /mysql.servers WHERE /; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + $data{$row->[1]}{owner} = $row->[0]; + $data{$row->[1]}{username} = $row->[2]; + $data{$row->[1]}{host} = $row->[3]; + $data{$row->[1]}{db} = $row->[4]; + $data{$row->[1]}{port} = $row->[5]; + $data{$row->[1]}{password} = $row->[6]; + } + + return %data; +} + +=head2 _get_partitions + +This function implements an MySQL-native partitions information. +Return two hash ref with partition details and partition default. +=cut + +sub _get_partitions +{ + my($self) = @_; + + # Retrieve all partitions. + my $str = qq{ +SELECT TABLE_NAME, PARTITION_ORDINAL_POSITION, PARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, PARTITION_METHOD, PARTITION_EXPRESSION +FROM INFORMATION_SCHEMA.PARTITIONS +WHERE PARTITION_NAME IS NOT NULL AND SUBPARTITION_NAME IS NULL AND (PARTITION_METHOD = 'RANGE' OR PARTITION_METHOD = 'LIST') +}; + $str .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|PARTITION_NAME'); + if ($self->{schema}) { + $str .= "\tAND TABLE_SCHEMA ='$self->{schema}'\n"; + } + $str .= "ORDER BY TABLE_NAME,PARTITION_ORDINAL_POSITION\n"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my %parts = (); + my %default = (); + while (my $row = $sth->fetch) { + if ( ($row->[3] eq 'MAXVALUE') || ($row->[3] eq 'DEFAULT')) { + $default{$row->[0]} = $row->[2]; + next; + } + my $i = $#{$parts{$row->[0]}{$row->[1]}{$row->[2]}} + 1; + push(@{$parts{$row->[0]}{$row->[1]}{$row->[2]}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $row->[6], 'colpos' => $i, 'tablespace' => $row->[4], 'owner' => ''}); + $self->logit(".",1); + } + $sth->finish; + $self->logit("\n", 1); + + return \%parts, \%default; +} + +=head2 _get_subpartitions + +This function implements a MySQL subpartitions information. +Return two hash ref with partition details and partition default. +=cut + +sub _get_subpartitions +{ + my($self) = @_; + + # Retrieve all partitions. + my $str = qq{ +SELECT TABLE_NAME, SUBPARTITION_ORDINAL_POSITION, SUBPARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, SUBPARTITION_METHOD, SUBPARTITION_EXPRESSION +FROM INFORMATION_SCHEMA.PARTITIONS +WHERE SUBPARTITION_NAME IS NOT NULL AND SUBPARTITION_EXPRESSION IS NOT NULL AND (SUBPARTITION_METHOD = 'RANGE' OR SUBPARTITION_METHOD = 'LIST') +}; + $str .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|PARTITION_NAME'); + if ($self->{schema}) { + $str .= " AND TABLE_SCHEMA ='$self->{schema}'\n"; + } + $str .= " ORDER BY TABLE_NAME,PARTITION_ORDINAL_POSITION\n"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + my %subparts = (); + my %default = (); + while (my $row = $sth->fetch) { + if ( ($row->[3] eq 'MAXVALUE') || ($row->[3] eq 'DEFAULT')) { + $default{$row->[0]} = $row->[2]; + next; + } + my $i = $#{$subparts{$row->[0]}{$row->[1]}{$row->[2]}} + 1; + push(@{$subparts{$row->[0]}{$row->[1]}{$row->[2]}}, { 'type' => $row->[5], 'value' => $row->[3], 'column' => $row->[6], 'colpos' => $i, 'tablespace' => $row->[4], 'owner' => ''}); + $self->logit(".",1); + } + $sth->finish; + $self->logit("\n", 1); + + return \%subparts, \%default; +} + +=head2 _get_partitions_list + +This function implements a MySQL-native partitions information. +Return a hash of the partition table_name => type + +=cut + +sub _get_partitions_list +{ + my($self) = @_; + + # Retrieve all partitions. + my $str = qq{ +SELECT TABLE_NAME, PARTITION_ORDINAL_POSITION, PARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, PARTITION_METHOD +FROM INFORMATION_SCHEMA.PARTITIONS WHERE SUBPARTITION_NAME IS NULL AND PARTITION_NAME IS NOT NULL +}; + $str .= $self->limit_to_objects('TABLE|PARTITION','TABLE_NAME|PARTITION_NAME'); + if ($self->{schema}) { + $str .= " AND TABLE_SCHEMA ='$self->{schema}'"; + } + $str .= " ORDER BY TABLE_NAME,PARTITION_NAME\n"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %parts = (); + while (my $row = $sth->fetch) { + $parts{$row->[5]}++; + } + $sth->finish; + + return %parts; +} + +=head2 _get_partitioned_table + +Return a hash of the partitioned table with the number of partition + +=cut + +sub _get_partitioned_table +{ + my($self) = @_; + + # Retrieve all partitions. + my $str = qq{ +SELECT TABLE_NAME, PARTITION_ORDINAL_POSITION, PARTITION_NAME, PARTITION_DESCRIPTION, TABLESPACE_NAME, PARTITION_METHOD +FROM INFORMATION_SCHEMA.PARTITIONS WHERE SUBPARTITION_NAME IS NULL AND PARTITION_NAME IS NOT NULL +}; + $str .= $self->limit_to_objects('TABLE|PARTITION','TABLE_NAME|PARTITION_NAME'); + if ($self->{schema}) { + $str .= " AND TABLE_SCHEMA ='$self->{schema}'"; + } + $str .= " ORDER BY TABLE_NAME,PARTITION_NAME\n"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %parts = (); + while (my $row = $sth->fetch) { + $parts{$row->[0]}++ if ($row->[2]); + } + $sth->finish; + + return %parts; +} + + +=head2 _get_objects + +This function retrieves all object the Oracle information + +=cut + +sub _get_objects +{ + my $self = shift; + + my %infos = (); + + # TABLE + my $sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$self->{schema}'"; + my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@{$infos{TABLE}}, { ( name => $row[0], invalid => 0) }); + } + $sth->finish(); + # VIEW + $sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.VIEWS WHERE TABLE_SCHEMA = '$self->{schema}'"; + $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@{$infos{VIEW}}, { ( name => $row[0], invalid => 0) }); + } + $sth->finish(); + # TRIGGER + $sql = "SELECT TRIGGER_NAME FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_SCHEMA = '$self->{schema}'"; + $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@{$infos{TRIGGER}}, { ( name => $row[0], invalid => 0) }); + } + $sth->finish(); + # INDEX + foreach my $t (@{$infos{TABLE}}) { + $sth = $self->{dbh}->prepare("SHOW INDEX FROM $t->{name} FROM $self->{schema}") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my @row = $sth->fetchrow()) { + next if ($row[2] eq 'PRIMARY'); + push(@{$infos{INDEX}}, { ( name => $row[2], invalid => 0) }); + } + } + # FUNCTION + $sql = "SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES WHERE DATA_TYPE IS NOT NULL AND ROUTINE_SCHEMA = '$self->{schema}'"; + if ($self->{db_version} lt '5.5.0') { + $sql =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; + } + $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@{$infos{FUNCTION}}, { ( name => $row[0], invalid => 0) }); + } + $sth->finish(); + # PROCEDURE + $sql = "SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES WHERE DATA_TYPE IS NULL AND ROUTINE_SCHEMA = '$self->{schema}'"; + if ($self->{db_version} lt '5.5.0') { + $sql =~ s/\bDATA_TYPE\b/DTD_IDENTIFIER/; + } + $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@{$infos{PROCEDURE}}, { ( name => $row[0], invalid => 0) }); + } + $sth->finish(); + + # PARTITION. + my $str = qq{ +SELECT TABLE_NAME||'_'||PARTITION_NAME +FROM INFORMATION_SCHEMA.PARTITIONS +WHERE SUBPARTITION_NAME IS NULL AND (PARTITION_METHOD = 'RANGE' OR PARTITION_METHOD = 'LIST') +}; + $sql .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|PARTITION_NAME'); + if ($self->{schema}) { + $sql .= "\tAND TABLE_SCHEMA ='$self->{schema}'\n"; + } + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@{$infos{'TABLE PARTITION'}}, { ( name => $row[0], invalid => 0) }); + } + $sth->finish; + + # SUBPARTITION. + $str = qq{ +SELECT TABLE_NAME||'_'||SUBPARTITION_NAME +FROM INFORMATION_SCHEMA.PARTITIONS +WHERE SUBPARTITION_NAME IS NOT NULL +}; + $sql .= $self->limit_to_objects('TABLE|PARTITION', 'TABLE_NAME|SUBPARTITION_NAME'); + if ($self->{schema}) { + $sql .= "\tAND TABLE_SCHEMA ='$self->{schema}'\n"; + } + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while ( my @row = $sth->fetchrow()) { + push(@{$infos{'TABLE PARTITION'}}, { ( name => $row[0], invalid => 0) }); + } + $sth->finish; + + return %infos; +} + +sub _get_privilege +{ + my($self) = @_; + + my %privs = (); + my %roles = (); + + # Retrieve all privilege per table defined in this database + my $str = "SELECT GRANTEE,TABLE_NAME,PRIVILEGE_TYPE,IS_GRANTABLE FROM INFORMATION_SCHEMA.TABLE_PRIVILEGES"; + if ($self->{schema}) { + $str .= " WHERE TABLE_SCHEMA = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'GRANTEE|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME'); + $str .= " ORDER BY TABLE_NAME, GRANTEE"; + my $error = "\n\nFATAL: You must be connected as an oracle dba user to retrieved grants\n\n"; + my $sth = $self->{dbh}->prepare($str) or $self->logit($error . "FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + # Remove the host part of the user + $row->[0] =~ s/\@.*//; + $row->[0] =~ s/'//g; + $privs{$row->[1]}{type} = $row->[2]; + if ($row->[3] eq 'YES') { + $privs{$row->[1]}{grantable} = $row->[3]; + } + $privs{$row->[1]}{owner} = ''; + push(@{$privs{$row->[1]}{privilege}{$row->[0]}}, $row->[2]); + push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); + } + $sth->finish(); + + # Retrieve all privilege per column table defined in this database + $str = "SELECT GRANTEE,TABLE_NAME,PRIVILEGE_TYPE,COLUMN_NAME,IS_GRANTABLE FROM INFORMATION_SCHEMA.COLUMN_PRIVILEGES"; + if ($self->{schema}) { + $str .= " WHERE TABLE_SCHEMA = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('GRANT|TABLE|VIEW|FUNCTION|PROCEDURE|SEQUENCE', 'GRANTEE|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME|TABLE_NAME'); + + $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + $row->[0] =~ s/\@.*//; + $row->[0] =~ s/'//g; + $privs{$row->[1]}{owner} = ''; + push(@{$privs{$row->[1]}{column}{$row->[3]}{$row->[0]}}, $row->[2]); + push(@{$roles{grantee}}, $row->[0]) if (!grep(/^$row->[0]$/, @{$roles{grantee}})); + } + $sth->finish(); + + return (\%privs, \%roles); +} + +=head2 _get_database_size + +This function retrieves the size of the MySQL database in MB + +=cut + +sub _get_database_size +{ + my $self = shift; + + my $mb_size = ''; + my $condition = ''; + + my $sql = qq{ +SELECT TABLE_SCHEMA "DB Name", + sum(DATA_LENGTH + INDEX_LENGTH)/1024/1024 "DB Size in MB" +FROM INFORMATION_SCHEMA.TABLES +WHERE TABLE_SCHEMA='$self->{schema}' +GROUP BY TABLE_SCHEMA +}; + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute or return undef; + while ( my @row = $sth->fetchrow()) { + $mb_size = sprintf("%.2f MB", $row[1]); + last; + } + $sth->finish(); + + return $mb_size; +} + +=head2 _get_largest_tables + +This function retrieves the list of largest table of the Oracle database in MB + +=cut + +sub _get_largest_tables +{ + my $self = shift; + + my %table_size = (); + + my $sql = qq{ +SELECT TABLE_NAME, sum(DATA_LENGTH + INDEX_LENGTH)/1024/1024 AS TSize +FROM INFORMATION_SCHEMA.TABLES +WHERE TABLE_SCHEMA='$self->{schema}' +}; + + $sql .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); + $sql .= " GROUP BY TABLE_NAME ORDER BY tsize"; + $sql .= " LIMIT $self->{top_max}" if ($self->{top_max}); + + my $sth = $self->{dbh}->prepare( $sql ) or return undef; + $sth->execute(@{$self->{query_bind_params}}) or return undef; + while ( my @row = $sth->fetchrow()) { + $table_size{$row[0]} = $row[1]; + } + $sth->finish(); + + return %table_size; +} + +sub _get_audit_queries +{ + my($self) = @_; + + return if (!$self->{audit_user}); + + my @users = (); + push(@users, split(/[,;\s]/, lc($self->{audit_user}))); + + # Retrieve all object with tablespaces. + my $str = "SELECT argument FROM mysql.general_log WHERE command_type='Query' AND argument REGEXP '^(INSERT|UPDATE|DELETE|SELECT)'"; + if (($#users >= 0) && !grep(/^all$/, @users)) { + $str .= " AND user_host REGEXP '(" . join("'|'", @users) . ")'"; + } + my $error = "\n\nFATAL: You must be connected as an oracle dba user to retrieved audited queries\n\n"; + my $sth = $self->{dbh}->prepare($str) or $self->logit($error . "FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %tmp_queries = (); + while (my $row = $sth->fetch) { + $self->_remove_comments(\$row->[0]); + $row->[0] = $self->normalize_query($row->[0]); + $tmp_queries{$row->[0]}++; + $self->logit(".",1); + } + $sth->finish; + $self->logit("\n", 1); + + my %queries = (); + my $i = 1; + foreach my $q (keys %tmp_queries) { + $queries{$i} = $q; + $i++; + } + + return %queries; +} + +sub _get_synonyms +{ + my ($self) = shift; + + return; +} + +sub _get_tablespaces +{ + my ($self) = shift; + + return; +} + +sub _list_tablespaces +{ + my ($self) = shift; + + return; +} + +sub _get_sequences +{ + my ($self) = shift; + + return; +} + +sub _extract_sequence_info +{ + my ($self) = shift; + + return; +} + +# MySQL does not have sequences but we count auto_increment as sequences +sub _count_sequences +{ + my $self = shift; + + # Table: information_schema.tables + # TABLE_CATALOG | varchar(512) | NO | | | | + # TABLE_SCHEMA | varchar(64) | NO | | | | + # TABLE_NAME | varchar(64) | NO | | | | + # TABLE_TYPE | varchar(64) | NO | | | | + # ENGINE | varchar(64) | YES | | NULL | | + # VERSION | bigint(21) unsigned | YES | | NULL | | + # ROW_FORMAT | varchar(10) | YES | | NULL | | + # TABLE_ROWS | bigint(21) unsigned | YES | | NULL | | + # AVG_ROW_LENGTH | bigint(21) unsigned | YES | | NULL | | + # DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | + # MAX_DATA_LENGTH | bigint(21) unsigned | YES | | NULL | | + # INDEX_LENGTH | bigint(21) unsigned | YES | | NULL | | + # DATA_FREE | bigint(21) unsigned | YES | | NULL | | + # AUTO_INCREMENT | bigint(21) unsigned | YES | | NULL | | + # CREATE_TIME | datetime | YES | | NULL | | + # UPDATE_TIME | datetime | YES | | NULL | | + # CHECK_TIME | datetime | YES | | NULL | | + # TABLE_COLLATION | varchar(32) | YES | | NULL | | + # CHECKSUM | bigint(21) unsigned | YES | | NULL | | + # CREATE_OPTIONS | varchar(255) | YES | | NULL | | + # TABLE_COMMENT | varchar(2048) | NO | | | | + + my @seqs = (); + my $sql = "SELECT TABLE_NAME, AUTO_INCREMENT FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA = '$self->{schema}'"; + $sql .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); + my $sth = $self->{dbh}->prepare( $sql ) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $row = $sth->fetch) { + push(@seqs, $row->[0]) if ($row->[1]); + } + $sth->finish(); + + return \@seqs; +} + +sub _column_attributes +{ + my ($self, $table, $owner, $objtype) = @_; + + $objtype ||= 'TABLE'; + + my $condition = ''; + if ($self->{schema}) { + $condition .= "AND TABLE_SCHEMA='$self->{schema}' "; + } + $condition .= "AND TABLE_NAME='$table' " if ($table); + if (!$table) { + $condition .= $self->limit_to_objects('TABLE', 'TABLE_NAME'); + } else { + @{$self->{query_bind_params}} = (); + } + $condition =~ s/^AND/WHERE/; + + # TABLE_CATALOG | varchar(512) | NO | | | | + # TABLE_SCHEMA | varchar(64) | NO | | | | + # TABLE_NAME | varchar(64) | NO | | | | + # COLUMN_NAME | varchar(64) | NO | | | | + # ORDINAL_POSITION | bigint(21) unsigned | NO | | 0 | | + # COLUMN_DEFAULT | longtext | YES | | NULL | | + # IS_NULLABLE | varchar(3) | NO | | | | + # DATA_TYPE | varchar(64) | NO | | | | + # CHARACTER_MAXIMUM_LENGTH | bigint(21) unsigned | YES | | NULL | | + # CHARACTER_OCTET_LENGTH | bigint(21) unsigned | YES | | NULL | | + # NUMERIC_PRECISION | bigint(21) unsigned | YES | | NULL | | + # NUMERIC_SCALE | bigint(21) unsigned | YES | | NULL | | + # CHARACTER_SET_NAME | varchar(32) | YES | | NULL | | + # COLLATION_NAME | varchar(32) | YES | | NULL | | + # COLUMN_TYPE | longtext | NO | | NULL | | + # COLUMN_KEY | varchar(3) | NO | | | | + # EXTRA | varchar(27) | NO | | | | + # PRIVILEGES | varchar(80) | NO | | | | + # COLUMN_COMMENT | varchar(1024) | NO | | | | + + my $sth = $self->{dbh}->prepare(<logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + } + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %data = (); + while (my $row = $sth->fetch) { + $data{$row->[3]}{"$row->[0]"}{nullable} = $row->[1]; + $data{$row->[3]}{"$row->[0]"}{default} = $row->[2]; + } + + return %data; + +} + +sub _list_triggers +{ + my($self) = @_; + + my $str = "SELECT TRIGGER_NAME, EVENT_OBJECT_TABLE FROM INFORMATION_SCHEMA.TRIGGERS"; + if ($self->{schema}) { + $str .= " AND TRIGGER_SCHEMA = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('TABLE|VIEW|TRIGGER','EVENT_OBJECT_TABLE|EVENT_OBJECT_TABLE|TRIGGER_NAME'); + $str =~ s/ AND / WHERE /; + + $str .= " ORDER BY EVENT_OBJECT_TABLE, TRIGGER_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %triggers = (); + while (my $row = $sth->fetch) { + push(@{$triggers{$row->[1]}}, $row->[0]); + } + + return %triggers; +} + +sub _global_temp_table_info +{ + my($self) = @_; + + return; +} + +sub _encrypted_columns +{ + my($self) = @_; + + return; +} + +sub _get_subpartitioned_table +{ + my($self) = @_; + + return; +} + +# Replace IF("user_status"=0,"username",NULL) +# PostgreSQL (CASE WHEN "user_status"=0 THEN "username" ELSE NULL END) +sub replace_if +{ + my $str = shift; + + # First remove all IN (...) before processing + my %in_clauses = (); + my $j = 0; + while ($str =~ s/\b(IN\s*\([^\(\)]+\))/,\%INCLAUSE$j\%/is) { + $in_clauses{$j} = $1; + $j++; + } + + while ($str =~ s/\bIF\s*\(((?:(?!\)\s*THEN|\s*SELECT\s+|\bIF\s*\().)*)$/\%IF\%$2/is || $str =~ s/\bIF\s*\(([^\(\)]+)\)(\s+AS\s+)/(\%IF\%)$2/is) { + my @if_params = (''); + my $stop_learning = 0; + my $idx = 1; + foreach my $c (split(//, $1)) { + $idx++ if (!$stop_learning && $c eq '('); + $idx-- if (!$stop_learning && $c eq ')'); + + if ($idx == 0) { + # Do not copy last parenthesis in the output string + $c = '' if (!$stop_learning); + # Inform the loop that we don't want to process any charater anymore + $stop_learning = 1; + # We have reach the end of the if() parameter + # next character must be restored to the final string. + $str .= $c; + } elsif ($idx > 0) { + # We are parsing the if() parameter part, append + # the caracter to the right part of the param array. + if ($c eq ',' && ($idx - 1) == 0) { + # we are switching to a new parameter + push(@if_params, ''); + } elsif ($c ne "\n") { + $if_params[-1] .= $c; + } + } + } + my $case_str = 'CASE '; + for (my $i = 1; $i <= $#if_params; $i+=2) { + $if_params[$i] =~ s/^\s+//gs; + $if_params[$i] =~ s/\s+$//gs; + if ($i < $#if_params) { + if ($if_params[$i] !~ /INCLAUSE/) { + $case_str .= "WHEN $if_params[0] THEN $if_params[$i] ELSE $if_params[$i+1] "; + } else { + $case_str .= "WHEN $if_params[0] $if_params[$i] THEN $if_params[$i+1] "; + } + } else { + $case_str .= " ELSE $if_params[$i] "; + } + } + $case_str .= 'END '; + + $str =~ s/\%IF\%/$case_str/s; + } + $str =~ s/\%INCLAUSE(\d+)\%/$in_clauses{$1}/gs; + $str =~ s/\s*,\s*IN\s*\(/ IN \(/igs; + + return $str; +} + +sub _get_plsql_metadata +{ + my $self = shift; + my $owner = shift; + + # Retrieve all functions + my $str = "SELECT ROUTINE_NAME,ROUTINE_SCHEMA,ROUTINE_TYPE,ROUTINE_DEFINITION FROM INFORMATION_SCHEMA.ROUTINES"; + if ($self->{schema}) { + $str .= " WHERE ROUTINE_SCHEMA = '$self->{schema}'"; + } + $str .= " ORDER BY ROUTINE_NAME"; + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + my %functions = (); + my @fct_done = (); + push(@fct_done, @EXCLUDED_FUNCTION); + while (my $row = $sth->fetch) { + next if (grep(/^$row->[0]$/i, @fct_done)); + push(@fct_done, "$row->[0]"); + $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{type} = $row->[2]; + $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{text} = $row->[3]; + my $sth2 = $self->{dbh}->prepare("SHOW CREATE $row->[2] $row->[0]") or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth2->execute or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + while (my $r = $sth2->fetch) { + $self->{function_metadata}{'unknown'}{'none'}{$row->[0]}{text} = $r->[2]; + last; + } + $sth2->finish(); + } + $sth->finish(); + + # Look for functions/procedures + foreach my $name (sort keys %{$self->{function_metadata}{'unknown'}{'none'}}) { + # Retrieve metadata for this function after removing comments + $self->_remove_comments(\$self->{function_metadata}{'unknown'}{'none'}{$name}{text}, 1); + $self->{comment_values} = (); + $self->{function_metadata}{'unknown'}{'none'}{$name}{text} =~ s/\%ORA2PG_COMMENT\d+\%//gs; + my %fct_detail = $self->_lookup_function($self->{function_metadata}{'unknown'}{'none'}{$name}{text}, $name); + if (!exists $fct_detail{name}) { + delete $self->{function_metadata}{'unknown'}{'none'}{$name}; + next; + } + delete $fct_detail{code}; + delete $fct_detail{before}; + %{$self->{function_metadata}{'unknown'}{'none'}{$name}{metadata}} = %fct_detail; + delete $self->{function_metadata}{'unknown'}{'none'}{$name}{text}; + } + +} + +sub _get_security_definer +{ + my ($self, $type) = @_; + + my %security = (); + + # Retrieve all functions security information + my $str = "SELECT ROUTINE_NAME,ROUTINE_SCHEMA,SECURITY_TYPE,DEFINER FROM INFORMATION_SCHEMA.ROUTINES"; + if ($self->{schema}) { + $str .= " WHERE ROUTINE_SCHEMA = '$self->{schema}'"; + } + $str .= " " . $self->limit_to_objects('FUNCTION|PROCEDURE', 'ROUTINE_NAME|ROUTINE_NAME'); + $str .= " ORDER BY ROUTINE_NAME"; + + my $sth = $self->{dbh}->prepare($str) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + $sth->execute(@{$self->{query_bind_params}}) or $self->logit("FATAL: " . $self->{dbh}->errstr . "\n", 0, 1); + + while (my $row = $sth->fetch) { + next if (!$row->[0]); + $security{$row->[0]}{security} = $row->[2]; + $security{$row->[0]}{owner} = $row->[3]; + } + $sth->finish(); + + return (\%security); +} + +=head2 _get_identities + +This function retrieve information about IDENTITY columns that must be +exported as PostgreSQL serial. + +=cut + +sub _get_identities +{ + my ($self) = @_; + + # nothing to do, AUTO_INCREMENT column are converted to serial/bigserial + return; +} + +=head2 _get_materialized_views + +This function implements a mysql-native materialized views information. + +Returns a hash of view names with the SQL queries they are based on. + +=cut + +sub _get_materialized_views +{ + my($self) = @_; + + # nothing to do, materialized view are not supported by MySQL. + return; +} + +1; + diff --git a/lib/Ora2Pg/PLSQL.pm b/lib/Ora2Pg/PLSQL.pm new file mode 100644 index 0000000000000000000000000000000000000000..b79def2db6356afeaf734b21e362e3e151201e9e --- /dev/null +++ b/lib/Ora2Pg/PLSQL.pm @@ -0,0 +1,3539 @@ +package Ora2Pg::PLSQL; +#------------------------------------------------------------------------------ +# Project : Oracle to PostgreSQL database schema converter +# Name : Ora2Pg/PLSQL.pm +# Language : Perl +# Authors : Gilles Darold, gilles _AT_ darold _DOT_ net +# Copyright: Copyright (c) 2000-2020 : Gilles Darold - All rights reserved - +# Function : Perl module used to convert Oracle PLSQL code into PL/PGSQL +# Usage : See documentation +#------------------------------------------------------------------------------ +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see < http://www.gnu.org/licenses/ >. +# +#------------------------------------------------------------------------------ + +use vars qw($VERSION %OBJECT_SCORE $SIZE_SCORE $FCT_TEST_SCORE $QUERY_TEST_SCORE %UNCOVERED_SCORE %UNCOVERED_MYSQL_SCORE @ORA_FUNCTIONS @MYSQL_SPATIAL_FCT @MYSQL_FUNCTIONS %EXCEPTION_MAP); +use POSIX qw(locale_h); + +#set locale to LC_NUMERIC C +setlocale(LC_NUMERIC,"C"); + + +$VERSION = '21.1'; + +#---------------------------------------------------- +# Cost scores used when converting PLSQL to PLPGSQL +#---------------------------------------------------- + +# Scores associated to each database objects: +%OBJECT_SCORE = ( + 'CLUSTER' => 0, # Not supported and no equivalent + 'FUNCTION' => 1, # read/adapt the header + 'INDEX' => 0.1, # Read/adapt - use varcharops like operator ? + 'FUNCTION-BASED-INDEX' => 0.2, # Check code of function call + 'REV-INDEX' => 1, # Check/rewrite the index to use trigram + 'CHECK' => 0.1, # Check/adapt the check constraint + 'MATERIALIZED VIEW' => 3, # Read/adapt, will just concern automatic snapshot export + 'PACKAGE BODY' => 3, # Look at globals variables and global + 'PROCEDURE' => 1, # read/adapt the header + 'SEQUENCE' => 0.1, # read/adapt to convert name.nextval() into nextval('name') + 'TABLE' => 0.1, # read/adapt the column type/name + 'TABLE PARTITION' => 0.1, # Read/check that table partitionning is ok + 'TABLE SUBPARTITION' => 0.2, # Read/check that table sub partitionning is ok + 'TRIGGER' => 1, # read/adapt the header + 'TYPE' => 1, # read + 'TYPE BODY' => 10, # Not directly supported need adaptation + 'VIEW' => 1, # read/adapt + 'DATABASE LINK' => 3, # Supported as FDW using oracle_fdw + 'GLOBAL TEMPORARY TABLE' => 10, # supported, but not permanent in PostgreSQL + 'DIMENSION' => 0, # Not supported and no equivalent + 'JOB' => 2, # read/adapt + 'SYNONYM' => 0.1, # read/adapt + 'QUERY' => 0.2, # read/adapt + 'ENCRYPTED COLUMN' => 20, ## adapt using pg_crypto +); + +# Scores following the number of characters: 1000 chars for one unit. +# Note: his correspond to the global read time not to the difficulty. +$SIZE_SCORE = 1000; + +# Cost to apply on each function or query for testing +$FCT_TEST_SCORE = 2; +$QUERY_TEST_SCORE = 0.1; + +# Scores associated to each code difficulties. +%UNCOVERED_SCORE = ( + 'TRUNC' => 0.1, + 'IS TABLE OF' => 4, + 'OUTER JOIN' => 2, + 'CONNECT BY' => 3, + 'BULK COLLECT' => 5, + 'GOTO' => 2, + 'FORALL' => 1, + 'ROWNUM' => 1, + 'NOTFOUND' => 0, + 'ISOPEN' => 1, + 'ROWCOUNT' => 1, + 'ROWID' => 2, + 'UROWID' => 2, + 'IS RECORD' => 1, + 'SQLCODE' => 1, + 'TABLE' => 2, + 'DBMS_' => 3, + 'DBMS_OUTPUT.put' => 1, + 'UTL_' => 3, + 'CTX_' => 3, + 'EXTRACT' => 0.1, + 'EXCEPTION' => 2, + 'TO_NUMBER' => 0.1, + 'REGEXP_LIKE' => 0.1, + 'REGEXP_COUNT' => 0.2, + 'REGEXP_INSTR' => 1, + 'REGEXP_SUBSTR' => 1, + 'TG_OP' => 0, + 'CURSOR' => 1, + 'PIPE ROW' => 1, + 'ORA_ROWSCN' => 3, + 'SAVEPOINT' => 1, + 'DBLINK' => 1, + 'PLVDATE' => 2, + 'PLVSTR' => 2, + 'PLVCHR' => 2, + 'PLVSUBST' => 2, + 'PLVLEX' => 2, + 'PLUNIT' => 2, + 'ADD_MONTHS' => 0.1, + 'LAST_DAY' => 1, + 'NEXT_DAY' => 1, + 'MONTHS_BETWEEN' => 1, + 'SDO_' => 3, + 'PRAGMA' => 3, + 'MDSYS' => 1, + 'MERGE INTO' => 3, + 'COMMIT' => 1, + 'CONTAINS' => 1, + 'SCORE' => 1, + 'FUZZY' => 1, + 'NEAR' => 1, + 'TO_CHAR' => 0.1, + 'TO_NCHAR' => 0.1, + 'ANYDATA' => 2, + 'CONCAT' => 0.1, + 'TIMEZONE' => 1, + 'JSON' => 3, + 'TO_CLOB' => 0.1 +); + +@ORA_FUNCTIONS = qw( + AsciiStr + Compose + Decompose + Dump + VSize + Bin_To_Num + CharToRowid + HexToRaw + NumToDSInterval + NumToYMInterval + RawToHex + To_Clob + To_DSInterval + To_Lob + To_Multi_Byte + To_NClob + To_Single_Byte + To_YMInterval + BFilename + Cardinality + Group_ID + LNNVL + NANVL + Sys_Context + Uid + UserEnv + Bin_To_Num + BitAnd + Cosh + Median + Remainder + Sinh + Tanh + DbTimeZone + New_Time + SessionTimeZone + Tz_Offset + Get_Env + From_Tz +); + +@MYSQL_SPATIAL_FCT = ( + 'AsBinary', + 'AsText', + 'Buffer', + 'Centroid', + 'Contains', + 'Crosses', + 'Dimension', + 'Disjoint', + 'EndPoint', + 'Envelope', + 'Equals', + 'ExteriorRing', + 'GeomCollFromText', + 'GeomCollFromWKB', + 'GeometryN', + 'GeometryType', + 'GeomFromText', + 'GeomFromWKB', + 'GLength', + 'InteriorRingN', + 'Intersects', + 'IsClosed', + 'IsSimple', + 'LineFromText', + 'LineFromWKB', + 'MLineFromText', + 'MPointFromText', + 'MPolyFromText', + 'NumGeometries', + 'NumInteriorRings', + 'NumPoints', + 'Overlaps', + 'Point', + 'PointFromText', + 'PointFromWKB', + 'PointN', + 'PolygonFromText', + 'Polygon', + 'SRID', + 'StartPoint', + 'Touches', + 'Within', + 'X', + 'Y' +); + +@MYSQL_FUNCTIONS = ( + 'AES_DECRYPT', + 'AES_ENCRYPT', + 'ASYMMETRIC_DECRYPT', + 'ASYMMETRIC_DERIVE', + 'ASYMMETRIC_ENCRYPT', + 'ASYMMETRIC_SIGN', + 'ASYMMETRIC_VERIFY', + 'CREATE_ASYMMETRIC_PRIV_KEY', + 'CREATE_ASYMMETRIC_PUB_KEY', + 'CREATE_DH_PARAMETERS', + 'CREATE_DIGEST', + 'DECODE', + 'DES_DECRYPT', + 'DES_ENCRYPT', + 'ENCODE', + 'ENCRYPT', + 'SHA1', + 'SHA2', + 'COLLATION', + 'COMPRESS', + 'CONVERT', + 'DEFAULT', + 'FOUND_ROWS', + 'GTID_SUBSET', + 'GTID_SUBTRACT', + 'INET6_ATON', + 'INET6_NTOA', + 'INTERVAL', + 'IS_FREE_LOCK', + 'IS_IPV4_COMPAT', + 'IS_IPV4_MAPPED', + 'IsEmpty', + 'LAST_INSERT_ID', + 'LOAD_FILE', + 'MASTER_POS_WAIT', + 'MATCH', + 'OLD_PASSWORD', + 'PERIOD_ADD', + 'PERIOD_DIFF', + 'RANDOM_BYTES', + 'ROW_COUNT', + 'SQL_THREAD_WAIT_AFTER_GTIDS', + 'WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS', + 'UNCOMPRESS', + 'UNCOMPRESSED_LENGTH', + 'UpdateXML', + 'UUID_SHORT', + 'VALIDATE_PASSWORD_STRENGTH', + 'WEIGHT_STRING', +); + +# Scores associated to each code difficulties after replacement. +%UNCOVERED_MYSQL_SCORE = ( + 'ARRAY_AGG_DISTINCT' => 1, # array_agg(distinct + 'SOUNDS LIKE' => 1, + 'CHARACTER SET' => 1, + 'COUNT(DISTINCT)' => 2, + 'MATCH' => 2, + 'JSON' => 2, + 'LOCK' => 2, + '@VAR' => 0.1, +); + +%EXCEPTION_MAP = ( + 'INVALID_CURSOR' => 'invalid_cursor_state', + 'ZERO_DIVIDE' => 'division_by_zero', + 'STORAGE_ERROR' => 'out_of_memory', + 'INTEGRITY_ERROR' => 'integrity_constraint_violation', + 'VALUE_ERROR' => 'data_exception', + 'INVALID_NUMBER' => 'data_exception', + 'INVALID_CURSOR' => 'invalid_cursor_state', + 'NO_DATA_FOUND' => 'no_data_found', + 'LOGIN_DENIED' => 'connection_exception', + 'TOO_MANY_ROWS'=> 'too_many_rows', + # 'PROGRAM_ERROR' => 'INTERNAL ERROR', + # 'ROWTYPE_MISMATCH' => 'DATATYPE MISMATCH' +); + + +=head1 NAME + +PSQL - Oracle to PostgreSQL procedural language converter + + +=head1 SYNOPSIS + + This external perl module is used to convert PLSQL code to PLPGSQL. + It is in an external code to allow easy editing and modification. + This converter is a work in progress and need your help. + + It is called internally by Ora2Pg.pm when you set PLSQL_PGSQL + configuration option to 1. +=cut + +=head2 convert_plsql_code + +Main function used to convert Oracle SQL and PL/SQL code into PostgreSQL +compatible code + +=cut + +sub convert_plsql_code +{ + my ($class, $str, @strings) = @_; + + return if ($str eq ''); + + # Replace outer join sign (+) with a placeholder + $class->{outerjoin_idx} //= 0; + while ( $str =~ s/\(\+\)/\%OUTERJOIN$class->{outerjoin_idx}\%/s ) { + $class->{outerjoin_idx}++; + } + + # Do some initialization of variables + %{$class->{single_fct_call}} = (); + $class->{replace_out_params} = ''; + + # Rewrite all decode() call before + $str = replace_decode($str) if (uc($class->{type}) ne 'SHOW_REPORT'); + + # Replace array syntax arr(i).x into arr[i].x + $str =~ s/\b([a-z0-9_]+)\(([^\(\)]+)\)(\.[a-z0-9_]+)/$1\[$2\]$3/igs; + + # Extract all block from the code by splitting it on the semi-comma + # character and replace all necessary function call + my @code_parts = split(/;/, $str); + for (my $i = 0; $i <= $#code_parts; $i++) + { + next if (!$code_parts[$i]); + + # For mysql also replace if() statements in queries or views. + if ($class->{is_mysql} && grep(/^$class->{type}$/i, 'VIEW', 'QUERY', 'FUNCTION', 'PROCEDURE')) { + $code_parts[$i] = Ora2Pg::MySQL::replace_if($code_parts[$i]); + } + + # Remove parenthesis from function parameters when they not belong to a function call + my %subparams = (); + my $p = 0; + while ($code_parts[$i] =~ s/(\(\s*)(\([^\(\)]*\))(\s*,)/$1\%SUBPARAMS$p\%$3/is) + { + $subparams{$p} = $2; + $p++; + } + while ($code_parts[$i] =~ s/(,\s*)(\([^\(\)]*\))(\s*[\),])/$1\%SUBPARAMS$p\%$3/is) + { + $subparams{$p} = $2; + $p++; + } + + # Remove some noisy parenthesis for outer join replacement + if ($code_parts[$i] =~ /\%OUTERJOIN\d+\%/) + { + my %tmp_ph = (); + my $idx = 0; + while ($code_parts[$i] =~ s/\(([^\(\)]*\%OUTERJOIN\d+\%[^\(\)]*)\)/\%SUBPART$idx\%/s) + { + $tmp_ph{$idx} = $1; + $idx++; + } + foreach my $k (keys %tmp_ph) + { + if ($tmp_ph{$k} =~ /^\s*[^\s]+\s*(=|NOT LIKE|LIKE)\s*[^\s]+\s*$/i) { + $code_parts[$i] =~ s/\%SUBPART$k\%/$tmp_ph{$k}/s; + } else { + $code_parts[$i] =~ s/\%SUBPART$k\%/\($tmp_ph{$k}\)/s; + } + } + } + + %{$class->{single_fct_call}} = (); + $code_parts[$i] = extract_function_code($class, $code_parts[$i], 0); + + # Things that must ne done when functions are replaced with placeholder + $code_parts[$i] = replace_without_function($class, $code_parts[$i]); + + foreach my $k (keys %{$class->{single_fct_call}}) + { + $class->{single_fct_call}{$k} = replace_oracle_function($class, $class->{single_fct_call}{$k}); + if ($class->{single_fct_call}{$k} =~ /^CAST\s*\(/i) + { + if (!$class->{is_mysql}) + { + $class->{single_fct_call}{$k} = Ora2Pg::PLSQL::replace_sql_type($class->{single_fct_call}{$k}, $class->{pg_numeric_type}, $class->{default_numeric}, $class->{pg_integer_type}, %{$class->{data_type}}); + } else { + $class->{single_fct_call}{$k} = Ora2Pg::MySQL::replace_sql_type($class->{single_fct_call}{$k}, $class->{pg_numeric_type}, $class->{default_numeric}, $class->{pg_integer_type}, %{$class->{data_type}}); + } + } + if ($class->{single_fct_call}{$k} =~ /^CAST\s*\(.*\%\%REPLACEFCT(\d+)\%\%/i) + { + if (!$class->{is_mysql}) { + $class->{single_fct_call}{$1} = Ora2Pg::PLSQL::replace_sql_type($class->{single_fct_call}{$1}, $class->{pg_numeric_type}, $class->{default_numeric}, $class->{pg_integer_type}, %{$class->{data_type}}); + } else { + $class->{single_fct_call}{$1} = Ora2Pg::MySQL::replace_sql_type($class->{single_fct_call}{$1}, $class->{pg_numeric_type}, $class->{default_numeric}, $class->{pg_integer_type}, %{$class->{data_type}}); + } + } + } + while ($code_parts[$i] =~ s/\%\%REPLACEFCT(\d+)\%\%/$class->{single_fct_call}{$1}/) {}; + $code_parts[$i] =~ s/\%SUBPARAMS(\d+)\%/$subparams{$1}/igs; + + } + $str = join(';', @code_parts); + + if ($class->{replace_out_params}) + { + if ($str !~ s/\b(DECLARE\s+)/$1$class->{replace_out_params}\n/is) { + $str =~ s/\b(BEGIN\s+)/DECLARE\n$class->{replace_out_params}\n$1/is; + } + $class->{replace_out_params} = ''; + } + + # Apply code rewrite on other part of the code + $str = plsql_to_plpgsql($class, $str, @strings); + + if ($class->{get_diagnostics}) + { + if ($str !~ s/\b(DECLARE\s+)/$1$class->{get_diagnostics}\n/is) { + $str =~ s/\b(BEGIN\s+)/DECLARE\n$class->{get_diagnostics}\n$1/is; + } + $class->{get_diagnostics} = ''; + } + + return $str; +} + +=head2 extract_function_code + +Recursive function used to extract call to function in Oracle SQL +and PL/SQL code + +=cut + +sub clear_parenthesis +{ + my $str = shift; + + # Keep parenthesys with sub queries + if ($str =~ /\bSELECT\b/i) { + $str = '((' . $str . '))'; + } else { + $str =~ s/^\s+//s; + $str =~ s/\s+$//s; + $str = '(' . $str . ')'; + } + + return $str; +} + +sub extract_function_code +{ + my ($class, $code, $idx) = @_; + + # Remove some extra parenthesis for better parsing + $code =~ s/\(\s*\(([^\(\)]*)\)\s*\)/clear_parenthesis($1)/iges; + + # Look for a function call that do not have an other function + # call inside, replace content with a marker and store the + # replaced string into a hask to rewritten later to convert pl/sql + if ($code =~ s/\b([a-zA-Z0-9\.\_]+)\s*\(([^\(\)]*)\)/\%\%REPLACEFCT$idx\%\%/s) { + my $fct_name = $1; + my $fct_code = $2; + my $space = ''; + $space = ' ' if (grep (/^$fct_name$/i, 'FROM', 'AS', 'VALUES', 'DEFAULT', 'OR', 'AND', 'IN', 'SELECT', 'OVER', 'WHERE', 'THEN', 'IF', 'ELSIF', 'ELSE', 'EXISTS', 'ON')); + + # Move up any outer join inside a function otherwise it will not be detected + my $outerjoin = ''; + if ($fct_code =~ /\%OUTERJOIN(\d+)\%/s) { + my $idx_join = $1; + # only if the placeholder content is a function not a predicate + if ($fct_code !~ /(=|>|<|LIKE|NULL|BETWEEN)/i) { + $fct_code =~ s/\%OUTERJOIN$idx_join\%//s; + $outerjoin = "\%OUTERJOIN$idx_join\%"; + } + } + # recursively replace function + $class->{single_fct_call}{$idx} = $fct_name . $space . '(' . $fct_code . ')' . $outerjoin; + $code = extract_function_code($class, $code, ++$idx); + } + + return $code; +} + +sub append_alias_clause +{ + my $str = shift; + + # Divise code through UNION keyword marking a new query level + my @q = split(/\b(UNION\s+ALL|UNION)\b/i, $str); + for (my $j = 0; $j <= $#q; $j+=2) { + if ($q[$j] =~ s/\b(FROM\s+)(.*\%SUBQUERY.*?)(\s*)(WHERE|ORDER\s+BY|GROUP\s+BY|LIMIT|$)/$1\%FROM_CLAUSE\%$3$4/is) { + my $from_clause = $2; + if ($q[$j] !~ /\b(YEAR|MONTH|DAY|HOUR|MINUTE|SECOND|TIMEZONE_HOUR|TIMEZONE_MINUTE|TIMEZONE_ABBR|TIMEZONE_REGION|TIMEZONE_OFFSET)\s+FROM/is) { + my @parts = split(/\b(WHERE|ORDER\s+BY|GROUP\s+BY|LIMIT)\b/i, $from_clause); + $parts[0] =~ s/(?=]|NOT LIKE|LIKE|WHERE|GROUP|ORDER)/is) { + $str =~ s/(END\b\s*)[\w"\.]+\s*(?:;|$)/$1;/is; + } + + return $str; +} + +=head2 set_error_code + +Transform custom exception code by replacing the leading -20 by 45 + +=cut + +sub set_error_code +{ + my $code = shift; + + my $orig_code = $code; + + $code =~ s/-20(\d{3})/'45$1'/; + if ($code =~ s/-20(\d{2})/'450$1'/ || $code =~ s/-20(\d{1})/'4500$1'/) { + print STDERR "WARNING: exception code has less than 5 digit, proceeding to automatic adjustement.\n"; + $code .= " /* code was: $orig_code */"; + } + + return $code; +} + + +=head2 plsql_to_plpgsql + +This function return a PLSQL code translated to PLPGSQL code + +=cut + +sub plsql_to_plpgsql +{ + my ($class, $str, @strings) = @_; + + return if ($str eq ''); + + return mysql_to_plpgsql($class, $str, @strings) if ($class->{is_mysql}); + + my $field = '\s*([^\(\),]+)\s*'; + my $num_field = '\s*([\d\.]+)\s*'; + my $date_field = '\s*([^,\)\(]*(?:date|time)[^,\)\(]*)\s*'; + + my $conv_current_time = 'clock_timestamp()'; + if (!grep(/$class->{type}/i, 'FUNCTION', 'PROCEDURE', 'PACKAGE')) { + $conv_current_time = 'LOCALTIMESTAMP'; + } + # Replace sysdate +/- N by localtimestamp - 1 day intervel + $str =~ s/\bSYSDATE\s*(\+|\-)\s*(\d+)/$conv_current_time $1 interval '$2 days'/igs; + + # Replace special case : (sysdate - to_date('01-Jan-1970', 'dd-Mon-yyyy'))*24*60*60 + # with: (extract(epoch from now()) + # When translating from code + while ($str =~ /\bSYSDATE\s*\-\s*to_date\(\s*\?TEXTVALUE(\d+)\?\s*,\s*\?TEXTVALUE(\d+)\?\s*\)\s*\)\s*\*\s*(24|60)\s*\*\s*(24|60)/is) { + my $t1 = $1; + my $t2 = $2; + if ($class->{text_values}{$t1} =~ /'(Jan|01).(Jan|01).1970'/ + && $class->{text_values}{$t2} =~ /'(Mon|MM|dd).(Mon|MM|dd).yyyy'/i) { + $str =~ s/\bSYSDATE\s*\-\s*to_date\(\s*\?TEXTVALUE(\d+)\?\s*,\s*\?TEXTVALUE(\d+)\?\s*\)\s*\)\s*\*\s*(24|60)\s*\*\s*(24|60)\*\s*(24|60)/extract(epoch from now()))/is; + } + } + + # When translating from default value (sysdate - to_date('01-01-1970','dd-MM-yyyy'))*24*60*60 + $str =~ s/\bSYSDATE\s*\-\s*to_date\(\s*'(Jan|01).(Jan|01).1970'\s*,\s*'(Mon|MM|dd).(Mon|MM|dd).yyyy'\s*\)\s*\)\s*\*\s*(24|60)\s*\*\s*(24|60)\s*\*\s*(24|60)/extract(epoch from now()))/igs; + + # Change SYSDATE to 'now' or current timestamp. + $str =~ s/\bSYSDATE\s*\(\s*\)/$conv_current_time/igs; + $str =~ s/\bSYSDATE\b/$conv_current_time/igs; + # Cast call to to_date with localtimestamp + $str =~ s/(TO_DATE\($conv_current_time)\s*,/$1::text,/igs; + + # JSON validation mostly in CHECK contraints + $str =~ s/((?:\w+\.)?\w+)\s+IS\s+JSON\b/\(CASE WHEN $1::json IS NULL THEN true ELSE true END\)/igs; + + # Drop temporary doesn't exist in PostgreSQL + $str =~ s/DROP\s+TEMPORARY/DROP/igs; + + # Private temporary table doesn't exist in PostgreSQL + $str =~ s/PRIVATE\s+TEMPORARY/TEMPORARY/igs; + $str =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION/ON COMMIT PRESERVE ROWS/igs; + $str =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION/ON COMMIT DROP/igs; + + # Replace SYSTIMESTAMP + $str =~ s/\bSYSTIMESTAMP\b/CURRENT_TIMESTAMP/igs; + # remove FROM DUAL + $str =~ s/FROM\s+DUAL//igs; + $str =~ s/FROM\s+SYS\.DUAL//igs; + + # DISTINCT and UNIQUE are synonym on Oracle + $str =~ s/SELECT\s+UNIQUE\s+([^,])/SELECT DISTINCT $1/igs; + + # Remove space between operators + $str =~ s/=\s+>/=>/gs; + $str =~ s/<\s+=/<=/gs; + $str =~ s/>\s+=/>=/gs; + $str =~ s/!\s+=/!=/gs; + $str =~ s/<\s+>/<>/gs; + $str =~ s/:\s+=/:=/gs; + $str =~ s/\|\s+\|/\|\|/gs; + $str =~ s/!=([+\-])/!= $1/gs; + + # replace operator for named parameters in function calls + if (!$class->{pg_supports_named_operator}) { + $str =~ s/([^<])=>/$1:=/gs; + } + + # Replace listagg() call + $str =~ s/\bLISTAGG\s*\((.*?)(?:\s*ON OVERFLOW [^\)]+)?\)\s+WITHIN\s+GROUP\s*\((.*?)\)/string_agg($1 $2)/ig; + + # There's no such things in PostgreSQL + $str =~ s/PRAGMA RESTRICT_REFERENCES[^;]+;//igs; + $str =~ s/PRAGMA SERIALLY_REUSABLE[^;]*;//igs; + $str =~ s/PRAGMA INLINE[^;]+;//igs; + + # Remove the extra TRUNCATE clauses not available in PostgreSQL + $str =~ s/TRUNCATE\s+TABLE\s+(.*?)\s+(REUSE|DROP)\s+STORAGE/TRUNCATE TABLE $1/igs; + $str =~ s/TRUNCATE\s+TABLE\s+(.*?)\s+(PRESERVE|PURGE)\s+MATERIALIZED\s+VIEW\s+LOG/TRUNCATE TABLE $1/igs; + + # Converting triggers + # :new. -> NEW. + $str =~ s/:new\./NEW\./igs; + # :old. -> OLD. + $str =~ s/:old\./OLD\./igs; + + # Change NVL to COALESCE + $str =~ s/NVL\s*\(/coalesce(/isg; + $str =~ s/NVL2\s*\($field,$field,$field\)/(CASE WHEN $1 IS NOT NULL THEN $2 ELSE $3 END)/isg; + + # NLSSORT to COLLATE + while ($str =~ /NLSSORT\($field,$field[\)]?/is) + { + my $col = $1; + my $nls_sort = $2; + if ($nls_sort =~ s/\%\%string(\d+)\%\%/$strings[$1]/s) { + $nls_sort =~ s/NLS_SORT=([^']+)[']*/COLLATE "$1"/is; + $nls_sort =~ s/\%\%ESCAPED_STRING\%\%//ig; + $str =~ s/NLSSORT\($field,$field[\)]?/$1 $nls_sort/is; + } elsif ($nls_sort =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/s) { + $nls_sort =~ s/\s*'NLS_SORT=([^']+)'/COLLATE "$1"/is; + $nls_sort =~ s/\%\%ESCAPED_STRING\%\%//ig; + $str =~ s/NLSSORT\($field,$field[\)]?/$1 $nls_sort/is; + } else { + $str =~ s/NLSSORT\($field,['\s]*NLS_SORT=([^']+)[']*/$1 COLLATE "$2"/is; + } + } + + # Replace EXEC function into variable, ex: EXEC :a := test(:r,1,2,3); + $str =~ s/\bEXEC\s+:([^\s:]+)\s*:=/SELECT INTO $2/igs; + + # Replace simple EXEC function call by SELECT function + $str =~ s/\bEXEC(\s+)/SELECT$1/igs; + + # Remove leading : on Oracle variable taking care of regex character class + $str =~ s/([^\w:]+):(\d+)/$1\$$2/igs; + $str =~ s/([^\w:]+):((?!alpha:|alnum:|blank:|cntrl:|digit:|graph:|lower:|print:|punct:|space:|upper:|xdigit:)\w+)/$1$2/igs; + + # INSERTING|DELETING|UPDATING -> TG_OP = 'INSERT'|'DELETE'|'UPDATE' + $str =~ s/\bINSERTING\b/TG_OP = 'INSERT'/igs; + $str =~ s/\bDELETING\b/TG_OP = 'DELETE'/igs; + $str =~ s/\bUPDATING\b/TG_OP = 'UPDATE'/igs; + # Replace Oracle call to column in trigger event + $str =~ s/TG_OP = '([^']+)'\s*\(\s*([^\)]+)\s*\)/TG_OP = '$1' AND NEW.$2 IS DISTINCT FROM OLD.$2/igs; + + # EXECUTE IMMEDIATE => EXECUTE + $str =~ s/EXECUTE IMMEDIATE/EXECUTE/igs; + + # SELECT without INTO should be PERFORM. Exclude select of view when prefixed with AS ot IS + if ( ($class->{type} ne 'QUERY') && ($class->{type} ne 'VIEW') ) { + $str =~ s/(\s+)(?{export_schema}) + { + if (!$class->{preserve_case}) + { + $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('\L$2\E')/isg; + $str =~ s/\b(\w+)\.(\w+)\.currval/currval('\L$2\E')/isg; + } + else + { + $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('"$2"')/isg; + $str =~ s/\b(\w+)\.(\w+)\.currval/currval('"$2"')/isg; + } + } + else + { + my $sch = $class->{pg_schema} || $class->{schema}; + if (!$class->{preserve_case}) + { + $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('\L$sch.$2\E')/isg; + $str =~ s/\b(\w+)\.(\w+)\.currval/currval('\L$sch.$2\E')/isg; + } + else + { + $str =~ s/\b(\w+)\.(\w+)\.nextval/nextval('"$sch"."$2"')/isg; + $str =~ s/\b(\w+)\.(\w+)\.currval/currval('"$sch"."$2"')/isg; + } + } + if (!$class->{preserve_case}) + { + $str =~ s/\b(\w+)\.nextval/nextval('\L$1\E')/isg; + $str =~ s/\b(\w+)\.currval/currval('\L$1\E')/isg; + } + else + { + $str =~ s/\b(\w+)\.nextval/nextval('"$1"')/isg; + $str =~ s/\b(\w+)\.currval/currval('"$1"')/isg; + } + + # Oracle MINUS can be replaced by EXCEPT as is + $str =~ s/\bMINUS\b/EXCEPT/igs; + # Comment DBMS_OUTPUT.ENABLE calls + $str =~ s/(DBMS_OUTPUT.ENABLE[^;]+;)/-- $1/isg; + # DBMS_LOB.GETLENGTH can be replaced by binary length. + $str =~ s/DBMS_LOB.GETLENGTH/octet_length/igs; + # DBMS_LOB.SUBSTR can be replaced by SUBSTR() + $str =~ s/DBMS_LOB.SUBSTR/substr/igs; + # TO_CLOB() + $str =~ s/TO_CLOB\(([^\)]+)\)/$1/igs; + + # Raise information to the client + $str =~ s/DBMS_OUTPUT\.(put_line|put|new_line)\s*\((.*?)\)\s*;/&raise_output($class, $2) . ';'/isge; + + # Simply remove this as not supported + $str =~ s/\bDEFAULT\s+NULL\b//igs; + + # Replace DEFAULT empty_blob() and empty_clob() + my $empty = "''"; + $empty = 'NULL' if ($class->{empty_lob_null}); + $str =~ s/(empty_blob|empty_clob)\s*\(\s*\)/$empty/is; + $str =~ s/(empty_blob|empty_clob)\b/$empty/is; + + # dup_val_on_index => unique_violation : already exist exception + $str =~ s/\bdup_val_on_index\b/unique_violation/igs; + + # Replace raise_application_error by PG standard RAISE EXCEPTION + $str =~ s/\braise_application_error\s*\(\s*([^,]+)\s*,\s*([^;]+),\s*(true|false)\s*\)\s*;/"RAISE EXCEPTION '%', $2 USING ERRCODE = " . set_error_code($1) . ";"/iges; + $str =~ s/\braise_application_error\s*\(\s*([^,]+)\s*,\s*([^;]+)\)\s*;/"RAISE EXCEPTION '%', $2 USING ERRCODE = " . set_error_code($1) . ";"/iges; + $str =~ s/DBMS_STANDARD\.RAISE EXCEPTION/RAISE EXCEPTION/igs; + + # Translate cursor declaration + $str = replace_cursor_def($str); + + # Remove remaining %ROWTYPE in other prototype declaration + #$str =~ s/\%ROWTYPE//isg; + + # Normalize HAVING ... GROUP BY into GROUP BY ... HAVING clause + $str =~ s/\bHAVING\b((?:(?!SELECT|INSERT|UPDATE|DELETE|WHERE|FROM).)*?)\bGROUP BY\b((?:(?!SELECT|INSERT|UPDATE|DELETE|WHERE|FROM).)*?)((?=UNION|ORDER BY|LIMIT|INTO |FOR UPDATE|PROCEDURE|\)\s+(?:AS)*[a-z0-9_]+\s+)|$)/GROUP BY$2 HAVING$1/gis; + + # Add STRICT keyword when select...into and an exception with NO_DATA_FOUND/TOO_MANY_ROW is present + #$str =~ s/\b(SELECT\b[^;]*?INTO)(.*?)(EXCEPTION.*?(?:NO_DATA_FOUND|TOO_MANY_ROW))/$1 STRICT $2 $3/igs; + # Add STRICT keyword when SELECT...INTO or EXECUTE ... INTO even if there's not EXCEPTION block + $str =~ s/\b((?:SELECT|EXECUTE)\s+[^;]*?\s+INTO)(\s+(?!STRICT))/$1 STRICT$2/igs; + $str =~ s/(INSERT\s+INTO\s+)STRICT\s+/$1/igs; + + # Remove the function name repetion at end + $str =~ s/\b(END\s*[^;\s]+\s*(?:;|$))/remove_fct_name($1)/iges; + + # Rewrite comment in CASE between WHEN and THEN + $str =~ s/(\s*)(WHEN\s+[^\s]+\s*)(\%ORA2PG_COMMENT\d+\%)(\s*THEN)/$1$3$1$2$4/igs; + + # Replace SQLCODE by SQLSTATE + $str =~ s/\bSQLCODE\b/SQLSTATE/igs; + + # Revert order in FOR IN REVERSE + $str =~ s/\bFOR(.*?)IN\s+REVERSE\s+([^\.\s]+)\s*\.\.\s*([^\s]+)/FOR$1IN REVERSE $3..$2/isg; + + # Comment call to COMMIT or ROLLBACK in the code if allowed + if ($class->{comment_commit_rollback}) { + $str =~ s/\b(COMMIT|ROLLBACK)\s*;/-- $1;/igs; + $str =~ s/(ROLLBACK\s+TO\s+[^;]+);/-- $1;/igs; + } + + # Comment call to SAVEPOINT in the code if allowed + if ($class->{comment_savepoint}) { + $str =~ s/(SAVEPOINT\s+[^;]+);/-- $1;/igs; + } + + # Replace exit at end of cursor + $str =~ s/EXIT\s+WHEN\s+([^\%;]+)\%\s*NOTFOUND\s*;/EXIT WHEN NOT FOUND; \/\* apply on $1 \*\//isg; + $str =~ s/EXIT\s+WHEN\s+\(\s*([^\%;]+)\%\s*NOTFOUND\s*\)\s*;/EXIT WHEN NOT FOUND; \/\* apply on $1 \*\//isg; + # Same but with additional conditions + $str =~ s/EXIT\s+WHEN\s+([^\%;]+)\%\s*NOTFOUND\s+([^;]+);/EXIT WHEN NOT FOUND $2; \/\* apply on $1 \*\//isg; + $str =~ s/EXIT\s+WHEN\s+\(\s*([^\%;]+)\%\s*NOTFOUND\s+([^\)]+)\)\s*;/EXIT WHEN NOT FOUND $2; \/\* apply on $1 \*\//isg; + # Replacle call to SQL%NOTFOUND and SQL%FOUND + $str =~ s/SQL\s*\%\s*NOTFOUND/NOT FOUND/isg; + $str =~ s/SQL\s*\%\s*FOUND/FOUND/isg; + + # Replace UTL_MATH function by fuzzymatch function + $str =~ s/UTL_MATCH.EDIT_DISTANCE/levenshtein/igs; + + # Replace known EXCEPTION equivalent ERROR code + foreach my $e (keys %EXCEPTION_MAP) { + $str =~ s/\b$e\b/$EXCEPTION_MAP{"\U$e\L"}/igs; + } + + # Replace special IEEE 754 values for not a number and infinity + $str =~ s/BINARY_(FLOAT|DOUBLE)_NAN/'NaN'/igs; + $str =~ s/([\-]*)BINARY_(FLOAT|DOUBLE)_INFINITY/'$1Infinity'/igs; + $str =~ s/'([\-]*)Inf'/'$1Infinity'/igs; + + # Replace PIPE ROW by RETURN NEXT + $str =~ s/PIPE\s+ROW\s*/RETURN NEXT /igs; + $str =~ s/(RETURN NEXT )\(([^\)]+)\)/$1$2/igs; + + # Convert all x <> NULL or x != NULL clauses to x IS NOT NULL. + $str =~ s/\s*(<>|\!=)\s*NULL/ IS NOT NULL/igs; + # Convert all x = NULL clauses to x IS NULL. + $str =~ s/(?!:)(.)=\s*NULL/$1 IS NULL/igs; + + # Add missing FROM clause in DELETE statements minus MERGE and FK ON DELETE + $str =~ s/(\bDELETE\s+)(?!FROM|WHERE|RESTRICT|CASCADE|NO ACTION)\b/$1FROM /igs; + + # Revert changes on update queries for IS NULL transaltion in the target list only + while ($str =~ s/\b(UPDATE\s+((?!WHERE|;).)*)\s+IS NULL/$1 = NULL/is) {}; + + # Rewrite all IF ... IS NULL with coalesce because for Oracle empty and NULL is the same + if ($class->{null_equal_empty}) { + # Form: column IS NULL + $str =~ s/([a-z0-9_\."]+)\s*IS NULL/coalesce($1::text, '') = ''/igs; + $str =~ s/([a-z0-9_\."]+)\s*IS NOT NULL/($1 IS NOT NULL AND $1::text <> '')/igs; + # Form: fct(expression) IS NULL + $str =~ s/([a-z0-9_\."]+\s*\([^\)\(]*\))\s*IS NULL/coalesce($1::text, '') = ''/igs; + $str =~ s/([a-z0-9_\."]+\s*\([^\)\(]*\))\s*IS NOT NULL/($1 IS NOT NULL AND ($1)::text <> '')/igs; + } + + # Replace type in sub block + if (!$class->{is_mysql}) { + $str =~ s/(BEGIN.*?DECLARE\s+)(.*?)(\s+BEGIN)/$1 . Ora2Pg::PLSQL::replace_sql_type($2, $class->{pg_numeric_type}, $class->{default_numeric}, $class->{pg_integer_type}, %{$class->{data_type}}) . $3/iges; + } else { + $str =~ s/(BEGIN.*?DECLARE\s+)(.*?)(\s+BEGIN)/$1 . Ora2Pg::MySQL::replace_sql_type($2, $class->{pg_numeric_type}, $class->{default_numeric}, $class->{pg_integer_type}, %{$class->{data_type}}) . $3/iges; + } + + # Remove any call to MDSYS schema in the code + $str =~ s/\bMDSYS\.//igs; + + # Oracle doesn't require parenthesis after VALUES, PostgreSQL has + # similar proprietary syntax but parenthesis are mandatory + $str =~ s/(INSERT\s+INTO\s+(?:.*?)\s+VALUES\s+)([^\(\)\s]+)\s*;/$1\($2.*\);/igs; + + # Replace some windows function issues with KEEP (DENSE_RANK FIRST ORDER BY ...) + $str =~ s/\b(MIN|MAX|SUM|AVG|COUNT|VARIANCE|STDDEV)\s*\(([^\)]+)\)\s+KEEP\s*\(DENSE_RANK\s+(FIRST|LAST)\s+(ORDER\s+BY\s+[^\)]+)\)\s*(OVER\s*\(PARTITION\s+BY\s+[^\)]+)\)/$3_VALUE($2) $5 $4)/igs; + + $class->{sub_queries} = (); + $class->{sub_queries_idx} = 0; + + #### + # Replace ending ROWNUM with LIMIT or row_number() and replace (+) outer join + #### + # Catch potential subquery first and replace rownum in subqueries + my @statements = split(/;/, $str); + for ( my $i = 0; $i <= $#statements; $i++ ) + { + # Remove any unecessary parenthesis in code + $statements[$i] = remove_extra_parenthesis($statements[$i]); + + $class->{sub_parts} = (); + $class->{sub_parts_idx} = 0; + extract_subpart($class, \$statements[$i]); + + # Translate all sub parts of the query before applying translation on the main query + foreach my $z (sort {$a <=> $b } keys %{$class->{sub_parts}}) + { + if ($class->{sub_parts}{$z} =~ /\S/is) + { + $class->{sub_parts}{$z} = translate_statement($class, $class->{sub_parts}{$z}, 1); + if ($class->{sub_parts}{$z} =~ /SELECT/is) + { + $class->{sub_parts}{$z} .= $class->{limit_clause}; + $class->{limit_clause} = ''; + } + # Try to append aliases of subqueries in the from clause + $class->{sub_parts}{$z} = append_alias_clause($class->{sub_parts}{$z}); + } + # If subpart is not empty after transformation + if ($class->{sub_parts}{$z} =~ /\S/is) + { + # add open and closed parenthesis + $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; + } + elsif ($statements[$i] !~ /\s+(WHERE|AND|OR)\s*\%SUBQUERY$z\%/is) + { + # otherwise do not report the empty parenthesis when this is not a function + $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; + } + } + + # Try to append aliases of subqueries in the from clause + $statements[$i] = append_alias_clause($statements[$i]); + + $statements[$i] .= $class->{limit_clause}; + $class->{limit_clause} = ''; + + # Apply translation on the full query + $statements[$i] = translate_statement($class, $statements[$i]); + + $statements[$i] .= $class->{limit_clause}; + $class->{limit_clause} = ''; + + # then restore subqueries code into the main query + while ($statements[$i] =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) {}; + + # Remove unnecessary offset to position 0 which is the default + $statements[$i] =~ s/\s+OFFSET 0//igs; + + } + + map { s/[ ]+([\r\n]+)/$1/s; } @statements; + map { s/[ ]+$//; } @statements; + $str = join(';', @statements); + + # Rewrite some garbadged resulting from the transformation + while ($str =~ s/(\s+AND)\s+AND\b/$1/is) {}; + while ($str =~ s/(\s+OR)\s+OR\b/$1/is) {}; + while ($str =~ s/\s+AND(\s+\%ORA2PG_COMMENT\d+\%\s+)+(AND)\b/$1$2/is) {}; + while ($str =~ s/\s+OR(\s+\%ORA2PG_COMMENT\d+\%\s+)+(OR)\b/$1$2/is) {}; + $str =~ s/\(\s*(AND|OR)\b/\(/igs; + $str =~ s/(\s+WHERE)\s+(AND|OR)\b/$1/igs; + $str =~ s/(\s+WHERE)(\s+\%ORA2PG_COMMENT\d+\%\s+)+(AND|OR)\b/$1$2/igs; + + # Attempt to remove some extra parenthesis in simple case only + $str = remove_extra_parenthesis($str); + + # Remove cast in partition range + $str =~ s/TIMESTAMP\s*('[^']+')/$1/igs; + + # Replace call to SQL%ROWCOUNT + $str =~ s/([^\s]+)\s*:=\s*SQL\%ROWCOUNT/GET DIAGNOSTICS $1 = ROW_COUNT/igs; + if ($str =~ s/(IF\s+)SQL\%ROWCOUNT/GET DIAGNOSTICS ora2pg_rowcount = ROW_COUNT;\n$1ora2pg_rowcount/igs) { + $class->{get_diagnostics} = 'ora2pg_rowcount int;'; + } + + # Sometime variable used in FOR ... IN SELECT loop is not declared + # Append its RECORD declaration in the DECLARE section. + my $tmp_code = $str; + while ($tmp_code =~ s/\bFOR\s+([^\s]+)\s+IN(.*?)LOOP//is) + { + my $varname = $1; + my $clause = $2; + my @code = split(/\bBEGIN\b/i, $str); + if ($code[0] !~ /\bDECLARE\s+.*\b$varname\s+/is) + { + # When the cursor is refereing to a statement, declare + # it as record otherwise it don't need to be replaced + if ($clause =~ /\bSELECT\b/is) + { + # append variable declaration to declare section + if ($str !~ s/\bDECLARE\b/DECLARE\n $varname RECORD;/is) + { + # No declare section + $str = "DECLARE\n $varname RECORD;\n" . $str; + } + } + } + } + + # Rewrite direct call to function without out parameters using PERFORM + $str = perform_replacement($class, $str); + + # Restore non converted outer join + $str =~ s/\%OUTERJOIN\d+\%/\(\+\)/igs; + + return $str; +} + +############## +# Rewrite direct call to function without out parameters using PERFORM +############## +sub perform_replacement +{ + my ($class, $str) = @_; + + if (uc($class->{type}) =~ /^(PACKAGE|FUNCTION|PROCEDURE|TRIGGER)$/) { + foreach my $sch ( keys %{ $class->{function_metadata} }) { + foreach my $p ( keys %{ $class->{function_metadata}{$sch} }) { + foreach my $k (keys %{$class->{function_metadata}{$sch}{$p}}) { + my $fct_name = $class->{function_metadata}{$sch}{$p}{$k}{metadata}{fct_name} || ''; + next if (!$fct_name); + next if ($p ne 'none' && $str !~ /\b$p\.$fct_name\b/is && $str !~ /(^|[^\.])\b$fct_name\b/is); + next if ($p eq 'none' && $str !~ /\b$fct_name\b/is); + if (!$class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout}) { + if ($sch ne 'unknown' and $str =~ /\b$sch.$k\b/is) { + # Look if we need to use PERFORM to call the function + $str =~ s/(BEGIN|LOOP|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/igs; + while ($str =~ s/(EXCEPTION(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/is) {}; + $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(PERFORM $sch\.$k);/$1\(\);/igs; + } elsif ($str =~ /\b$k\b/is) { + # Look if we need to use PERFORM to call the function + $str =~ s/(BEGIN|LOOP|CALL|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/igs; + while ($str =~ s/(EXCEPTION(?:(?!CASE).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/is) {}; + $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/isg; + #$str =~ s/(WHEN(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(PERFORM $k);/$1\(\);/igs; + } else { + # Look if we need to use PERFORM to call the function + $str =~ s/(BEGIN|LOOP|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)($fct_name\s*[\(;])/$1$2PERFORM $3/igs; + while ($str =~ s/(EXCEPTION(?:(?!CASE).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($fct_name\s*[\(;])/$1$2PERFORM $3/is) {}; + $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($fct_name\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($fct_name\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(WHEN(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($fct_name\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(WHEN(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($fct_name\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(PERFORM $fct_name);/$1\(\);/igs; + } + } else { + # Recover call to function with OUT parameter with double affectation + $str =~ s/([^:\s]+\s*:=\s*)[^:\s]*\s+:=\s*((?:[^\s\.]+\.)?\b$fct_name\s*\()/$1$2/isg; + } + # Remove package name and try to replace call to function name only + if (!$class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout} && $k =~ s/^[^\.]+\.// && lc($p) eq lc($class->{current_package}) ) { + if ($sch ne 'unknown' and $str =~ /\b$sch\.$k\b/is) { + $str =~ s/(BEGIN|LOOP|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/igs; + while ($str =~ s/(EXCEPTION(?:(?!CASE).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/is) {}; + $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($sch\.$k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(PERFORM $sch\.$k);/$1\(\);/igs; + } elsif ($str =~ /\b$k\b/is) { + $str =~ s/(BEGIN|LOOP|CALL|;)((?:\s*%ORA2PG_COMMENT\d+\%\s*|\s*\/\*(?:.*?)\*\/\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/igs; + while ($str =~ s/(EXCEPTION(?:(?!CASE).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/is) {}; + $str =~ s/(IF(?:(?!CASE|THEN).)*?THEN)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(IF(?:(?!CASE|ELSE).)*?ELSE)((?:\s*%ORA2PG_COMMENT\d+\%\s*)*\s*)($k\s*[\(;])/$1$2PERFORM $3/isg; + $str =~ s/(PERFORM $k);/$1\(\);/igs; + } + } + } + } + } + } + + # Fix call to procedure changed above + if ($class->{pg_supports_procedure}) { + $str =~ s/\bCALL\s+PERFORM/CALL/igs; + } else { + $str =~ s/\bCALL\s+PERFORM/PERFORM/igs; + } + + return $str; +} + +sub translate_statement +{ + my ($class, $stmt, $is_subpart) = @_; + + # Divise code through UNION keyword marking a new query level + my @q = split(/\b(UNION\s+ALL|UNION)\b/i, $stmt); + for (my $j = 0; $j <= $#q; $j++) { + next if ($q[$j] =~ /^UNION/); + + # Replace call to right outer join obsolete syntax + $q[$j] = replace_outer_join($class, $q[$j], 'right'); + + # Replace call to left outer join obsolete syntax + $q[$j] = replace_outer_join($class, $q[$j], 'left'); + + if ($q[$j] =~ /\bROWNUM\b/i) + { + # Replace ROWNUM after the WHERE clause by a LIMIT clause + $q[$j] = replace_rownum_with_limit($class, $q[$j]); + # Replace ROWNUM by row_number() when used in the target list + $q[$j] =~ s/((?!WHERE\s.*|LIMIT\s.*)[\s,]+)ROWNUM([\s,]+)/$1row_number() OVER () AS rownum$2/is; + # Aliases before + or - will generate an error + $q[$j] =~ s/row_number\(\) OVER \(\) AS rownum\s*([+\-])/row_number() OVER () $1/is; + # Try to replace AS rownnum with alias if there is one already defined + $q[$j] =~ s/(row_number\(\) OVER \(\) AS)\s+rownum\s+((?!FROM\s+|[,+\-]\s*)[^\s]+)/$1 $2/is; + $q[$j] =~ s/\s+AS(\s+AS\s+)/$1/is; + # The form "UPDATE mytbl SET col1 = ROWNUM;" is not yet translated + # and mus be manually rewritten as follow: + # WITH cte AS (SELECT *, ROW_NUMBER() OVER() AS rn FROM mytbl) + # UPDATE mytbl SET col1 = (SELECT rn FROM cte WHERE cte.pk = mytbl.pk); + } + + } + $stmt = join("\n", @q); + + # Rewrite some invalid ending form after rewriting + $stmt =~ s/(\s+WHERE)\s+AND/$1/igs; + + $stmt =~ s/(\s+)(?:WHERE|AND)\s+(LIMIT\s+)/$1$2/igs; + $stmt =~ s/\s+WHERE\s*$//is; + $stmt =~ s/\s+WHERE\s*\)/\)/is; + + # Remove unnecessary offset to position 0 which is the default + $stmt =~ s/\s+OFFSET 0//igs; + + # Replacement of connect by with CTE + $stmt = replace_connect_by($class, $stmt); + + return $stmt; +} + +sub remove_extra_parenthesis +{ + my $str = shift; + + while ($str =~ s/\(\s*\(((?!\s*SELECT)[^\(\)]+)\)\s*\)/($1)/gs) {}; + my %store_clause = (); + my $i = 0; + while ($str =~ s/\(\s*\(([^\(\)]+)\)\s*AND\s*\(([^\(\)]+)\)\s*\)/\%PARENTHESIS$i\%/is) { + $store_clause{$i} = find_or_parenthesis($1, $2); + $i++ + } + $str =~ s/\%PARENTHESIS(\d+)\%/$store_clause{$1}/gs; + while ($str =~ s/\(\s*\(\s*\(([^\(\)]+\)[^\(\)]+\([^\(\)]+)\)\s*\)\s*\)/(($1))/gs) {}; + + return $str; +} + +# When the statement include OR keep parenthesisœ +sub find_or_parenthesis +{ + my ($left, $right) = @_; + + if ($left =~ /\s+OR\s+/i) { + $left = "($left)"; + } + if ($right =~ /\s+OR\s+/i) { + $right = "($right)"; + } + + return "($left AND $right)"; +} + + +sub extract_subpart +{ + my ($class, $str) = @_; + + while ($$str =~ s/\(([^\(\)]*)\)/\%SUBQUERY$class->{sub_parts_idx}\%/s) { + $class->{sub_parts}{$class->{sub_parts_idx}} = $1; + $class->{sub_parts_idx}++; + } + my @done = (); + foreach my $k (sort { $b <=> $a } %{$class->{sub_parts}}) { + if ($class->{sub_parts}{$k} =~ /\%OUTERJOIN\d+\%/ && $class->{sub_parts}{$k} !~ /\b(SELECT|FROM|WHERE)\b/i) { + $$str =~ s/\%SUBQUERY$k\%/\($class->{sub_parts}{$k}\)/s; + push(@done, $k); + } + } + foreach (@done) { + delete $class->{sub_parts}{$_}; + } +} + + +sub extract_subqueries +{ + my ($class, $str) = @_; + + return if ($class->{sub_queries_idx} == 100); + + my $cur_idx = $class->{sub_queries_idx}; + if ($$str =~ s/\((\s*(?:SELECT|WITH).*)/\%SUBQUERY$class->{sub_queries_idx}\%/is) { + my $stop_learning = 0; + my $idx = 1; + my $sub_query = ''; + foreach my $c (split(//, $1)) { + $idx++ if (!$stop_learning && $c eq '('); + $idx-- if (!$stop_learning && $c eq ')'); + if ($idx == 0) { + # Do not copy last parenthesis in the output string + $c = '' if (!$stop_learning); + # Increment storage position for the next subquery + $class->{sub_queries_idx}++ if (!$stop_learning); + # Inform the loop that we don't want to process any charater anymore + $stop_learning = 1; + # We have reach the end of the subquery all next + # characters must be restored to the final string. + $$str .= $c; + } elsif ($idx > 0) { + # Append character to the current substring storage + $class->{sub_queries}{$class->{sub_queries_idx}} .= $c; + } + } + + # Each subquery could have subqueries too, so call the + # function recursively on each extracted subquery + if ($class->{sub_queries}{$class->{sub_queries_idx}-1} =~ /\(\s*(?:SELECT|WITH)/is) { + extract_subqueries($class, \$class->{sub_queries}{$class->{sub_queries_idx}-1}); + } + } + +} + +sub replace_rownum_with_limit +{ + my ($class, $str) = @_; + + my $offset = ''; + if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { + $offset = $2; + ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; + $class->{limit_clause} = ' LIMIT 1 OFFSET ' . $offset; + + } + if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { + $offset = $1; + ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; + $class->{limit_clause} = ' LIMIT 1 OFFSET ' . $offset; + } + + if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*>=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { + $offset = $2; + ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; + $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; + } + if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*>\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { + $offset = $2; + $offset = "($offset)" if ($offset =~ /[^0-9]/); + $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; + } + if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*>=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { + $offset = $1; + ($offset =~ /[^0-9]/) ? $offset = "($offset)" : $offset -= 1; + $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; + } + if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*>\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { + $offset = $1; + $offset = "($offset)" if ($offset =~ /[^0-9]/); + $class->{limit_clause} = ' LIMIT ALL OFFSET ' . $offset; + } + + my $tmp_val = ''; + if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*<=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { + $tmp_val = $2; + } + if ($str =~ s/\s+(WHERE)\s+(?:\(\s*)?ROWNUM\s*<\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $1 $3$4/is) { + $tmp_val = $2 - 1; + } + if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*<=\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { + $tmp_val = $1; + } + if ($str =~ s/\s+AND\s+(?:\(\s*)?ROWNUM\s*<\s*([^\s\)]+)(\s*\)\s*)?([^;]*)/ $2$3/is) { + $tmp_val = $1 - 1; + } + + if ($tmp_val) { + if ($class->{limit_clause} =~ /LIMIT ALL OFFSET ([^\s]+)/is) { + my $tmp_offset = $1; + if ($tmp_offset !~ /[^0-9]/ && $tmp_val !~ /[^0-9]/) { + $tmp_val -= $tmp_offset; + } else { + $tmp_val = "($tmp_val - $tmp_offset)"; + } + $class->{limit_clause} =~ s/LIMIT ALL/LIMIT $tmp_val/is; + } else { + $tmp_val = "($tmp_val)" if ($tmp_val =~ /[^0-9]/); + $class->{limit_clause} = ' LIMIT ' . $tmp_val; + } + } + + # Rewrite some invalid ending form after rewriting + $str =~ s/(\s+WHERE)\s+AND/$1/igs; + $str =~ s/\s+WHERE\s*$//is; + $str =~ s/\s+WHERE\s*\)/\)/is; + + # Remove unnecessary offset to position 0 which is the default + $str =~ s/\s+OFFSET 0//igs; + + return $str; +} + +# Translation of REGEX_SUBSTR( string, pattern, [pos], [nth]) converted into +# (SELECT array_to_string(a, '') FROM regexp_matches(substr(string, pos), pattern, 'g') AS foo(a) LIMIT 1 OFFSET (nth - 1))"; +# Optional fith parameter of match_parameter is appended to 'g' when present +sub convert_regex_substr +{ + ($class, $str) = @_; + + my @params = split(/\s*,\s*/, $str); + my $mod = ''; + if ($#params == 4) { + # Restore constant string to look into date format + while ($params[4] =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/is) { + delete $class->{text_values}{$1}; + } + $params[4] =~ s/'//g; + $mod = $params[4] if ($params[4] ne 'g'); + } + if ($#params < 2) { + push(@params, 1, 1); + } elsif ($#params < 3) { + push(@params, 1); + } + if ($params[2] == 1) { + $str = "(SELECT array_to_string(a, '') FROM regexp_matches($params[0], $params[1], 'g$mod') AS foo(a) LIMIT 1 OFFSET ($params[3] - 1))"; + } else { + $str = "(SELECT array_to_string(a, '') FROM regexp_matches(substr($params[0], $params[2]), $params[1], 'g$mod') AS foo(a) LIMIT 1 OFFSET ($params[3] - 1))"; + } + + return $str; +} + +sub convert_from_tz +{ + my ($class, $date) = @_; + + # Restore constant string to look into date format + while ($date =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/is) { + delete $class->{text_values}{$1}; + } + + my $tz = '00:00'; + if ($date =~ /^[^']*'([^']+)'\s*,\s*'([^']+)'/) { + $date = $1; + $tz = $2; + $date = $date . ' '; + if ($tz =~ /^\d+:\d+$/) { + $date .= '+' . $tz; + } else { + $date .= $tz; + } + $date = "'$date'"; + } elsif ($date =~ /^(.*),\s*'([^']+)'$/) { + $date = $1; + $tz = $2; + if ($tz =~ /^\d+:\d+$/) { + $tz .= '+' . $tz; + } + $date = $date . ' AT TIME ZONE ' . "'$tz'"; + } + + # Replace constant strings + while ($date =~ s/('[^']+')/\?TEXTVALUE$class->{text_values_pos}\?/s) { + $class->{text_values}{$class->{text_values_pos}} = $1; + $class->{text_values_pos}++; + } + + return $date; +} + +sub convert_date_format +{ + my ($class, $fields) = @_; + + # Restore constant string to look into date format + while ($fields =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/is) { + delete $class->{text_values}{$1}; + } + + # Truncate time to microsecond + $fields =~ s/(\d{2}:\d{2}:\d{2}[,\.]\d{6})\d{3}/$1/s; + + # Replace round year with two digit year format. + $fields =~ s/RR/YY/sg; + + # Convert fractional seconds to milli (MS) or micro (US) seconds + $fields =~ s/FF[123]/MS/s; + $fields =~ s/FF\d*/US/s; + + # Remove any timezone format + $fields =~ s/\s*TZ[DHMR]//gs; + + # Replace constant strings + while ($str =~ s/('[^']+')/\?TEXTVALUE$class->{text_values_pos}\?/s) { + $class->{text_values}{$class->{text_values_pos}} = $1; + $class->{text_values_pos}++; + } + return $fields; +} + + +#------------------------------------------------------------------------------ +# Set the correspondance between Oracle and PostgreSQL regexp modifiers +# Oracle default: +# 1) The default case sensitivity is determined by the NLS_SORT parameter. +# Ora2pg assuming case sensitivy +# 2) A period (.) does not match the newline character. +# 3) The source string is treated as a single line. +# PostgreSQL default: +# 1) Default to case sensitivity +# 2) A period match the newline character. +# 3) The source string is treated as a single line. +# Oracle only supports the following modifiers +# 'i' specifies case-insensitive matching. Same for PG. +# 'c' specifies case-sensitive matching. Same for PG. +# 'x' Ignores whitespace characters in the search pattern. Same for PG. +# 'n' allows the period (.) to match the newline character. PG => s. +# 'm' treats the source string as multiple lines. PG => n. +#------------------------------------------------------------------------------ +sub regex_flags +{ + my ($class, $modifier, $append) = @_; + + my $nconst = ''; + my $flags = $append || ''; + + if ($modifier =~ /\?TEXTVALUE(\d+)\?/) + { + $nconst = $1; + $modifier =~ s/\?TEXTVALUE$nconst\?/$class->{text_values}{$nconst}/; + } + # These flags have the same behavior + if ($modifier =~ /([icx]+)/) { + $flags .= $1; + } + # Oracle: + # m : treats the source string as multiple lines. + # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('Hello'||CHR(10)||'world!', '^world!$', 'm'); => 1 + # PostgreSQL: + # m : historical synonym for n => m : newline-sensitive matching + # SELECT regexp_match('Hello'||chr(10)||'world!', '^world!$', 'm'); => match + if ($modifier =~ /m/) { + $flags .= 'n'; + } + # Oracle: + # n: allows the period (.) to match the newline character. + # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('a'||CHR(10)||'d', 'a.d', 'n'); => 1 + # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('a'||CHR(10)||'d', '^d$', 'n'); => not match + # PostgreSQL: + # s: non-newline-sensitive matching (default) + # SELECT regexp_match('a'||chr(10)||'d', 'a.d', 's'); => match + # SELECT regexp_match('a'||chr(10)||'d', '^d$', 's'); => not match + if ($modifier =~ /n/) { + $flags .= 's'; + } + + # By default PG is non-newline-sensitive whereas Oracle is newline-sensitive + # Oracle: + # SELECT '1' FROM DUAL WHERE REGEXP_LIKE('a'||CHR(10)||'d', 'a.d'); => not match + # PostgreSQL: + # SELECT regexp_match('a'||chr(10)||'d', 'a.d'); => match + # Add 'n' to force the same behavior like Oracle + $flags .= 'n' if ($flags !~ /n|s/); + + if ($nconst ne '') + { + $class->{text_values}{$nconst} = "'$flags'"; + return "?TEXTVALUE$nconst?"; + } + + return "'$flags'"; +} + +sub replace_oracle_function +{ + my ($class, $str) = @_; + + my @xmlelt = (); + my $field = '\s*([^\(\),]+)\s*'; + my $num_field = '\s*([\d\.]+)\s*'; + my $date_field = '\s*([^,\)\(]*(?:date|time)[^,\)\(]*)\s*'; + + #-------------------------------------------- + # PL/SQL to PL/PGSQL code conversion + # Feel free to add your contribution here. + #-------------------------------------------- + + if ($class->{is_mysql}) { + $str = mysql_to_plpgsql($class, $str); + } + + # Change NVL to COALESCE + $str =~ s/NVL\s*\(/coalesce(/is; + $str =~ s/NVL2\s*\($field,$field,$field\)/(CASE WHEN $1 IS NOT NULL THEN $2 ELSE $3 END)/is; + + # Replace DEFAULT empty_blob() and empty_clob() + my $empty = "''"; + $empty = 'NULL' if ($class->{empty_lob_null}); + $str =~ s/(empty_blob|empty_clob)\s*\(\s*\)/$empty/is; + $str =~ s/(empty_blob|empty_clob)\b/$empty/is; + + # DBMS_LOB.GETLENGTH can be replaced by binary length. + $str =~ s/DBMS_LOB.GETLENGTH/octet_length/igs; + # DBMS_LOB.SUBSTR can be replaced by SUBSTR() + $str =~ s/DBMS_LOB.SUBSTR/substr/igs; + # TO_CLOB() + $str =~ s/TO_CLOB\(([^\)]+)\)/$1/igs; + + # Replace call to SYS_GUID() function + $str =~ s/\bSYS_GUID\s*\(\s*\)/$class->{uuid_function}()/is; + $str =~ s/\bSYS_GUID\b/$class->{uuid_function}()/is; + + # Rewrite TO_DATE formating call + $str =~ s/TO_DATE\s*\(\s*('[^\']+')\s*,\s*('[^\']+')[^\)]*\)/to_date($1,$2)/is; + + # When the date format is ISO and we have a constant we can remove the call to to_date() + if ($class->{type} eq 'PARTITION' && $class->{pg_supports_partition}) { + $str =~ s/to_date\(\s*('\s*\d+-\d+-\d+ \d+:\d+:\d+')\s*,\s*'[S]*YYYY-MM-DD HH24:MI:SS'[^\)]*\)/$1/; + } + + # Translate to_timestamp_tz Oracle function + $str =~ s/TO_TIMESTAMP_TZ\s*\((.*)\)/'to_timestamp(' . convert_date_format($class, $1) . ')'/ies; + + # Translate from_tz Oracle function + $str =~ s/FROM_TZ\s*\(\s*([^\)]+)\s*\)/'(' . convert_from_tz($class,$1) . ')::timestamp with time zone'/ies; + + # Replace call to trim into btrim + $str =~ s/\bTRIM\s*\(([^\(\)]+)\)/trim(both $1)/is; + + # Do some transformation when Orafce is not used + if (!$class->{use_orafce}) + { + # Replace to_nchar() without format by a simple cast to text + $str =~ s/\bTO_NCHAR\s*\(\s*([^,\)]+)\)/($1)::varchar/igs; + # Replace to_char() without format by a simple cast to text + $str =~ s/\bTO_CHAR\s*\(\s*([^,\)]+)\)/($1)::varchar/igs; + if ($class->{type} ne 'TABLE') { + $str =~ s/\(([^\s]+)\)(::varchar)/$1$2/igs; + } else { + $str =~ s/\(([^\s]+)\)(::varchar)/($1$2)/igs; + } + + # Change trunc() to date_trunc('day', field) + # Trunc is replaced with date_trunc if we find date in the name of + # the value because Oracle have the same trunc function on number + # and date type + $str =~ s/\bTRUNC\s*\($date_field\)/date_trunc('day', $1)/is; + if ($str =~ s/\bTRUNC\s*\($date_field,$field\)/date_trunc($2, $1)/is || + # Case where the parameters are obfuscated by function and string placeholders + $str =~ s/\bTRUNC\((\%\%REPLACEFCT\d+\%\%)\s*,\s*(\?TEXTVALUE\d+\?)\)/date_trunc($2, $1)/is + ) + { + if ($str =~ /date_trunc\(\?TEXTVALUE(\d+)\?/) + { + my $k = $1; + $class->{text_values}{$k} =~ s/'(SYYYY|SYEAR|YEAR|[Y]+)'/'year'/is; + $class->{text_values}{$k} =~ s/'Q'/'quarter'/is; + $class->{text_values}{$k} =~ s/'(MONTH|MON|MM|RM)'/'month'/is; + $class->{text_values}{$k} =~ s/'(IW|DAY|DY|D)'/'week'/is; + $class->{text_values}{$k} =~ s/'(DDD|DD|J)'/'day'/is; + $class->{text_values}{$k} =~ s/'(HH|HH12|HH24)'/'hour'/is; + $class->{text_values}{$k} =~ s/'MI'/'minute'/is; + } + } + + # Convert the call to the Oracle function add_months() into Pg syntax + $str =~ s/ADD_MONTHS\s*\(([^,]+),\s*(\d+)\s*\)/$1 + '$2 month'::interval/si; + $str =~ s/ADD_MONTHS\s*\(([^,]+),\s*([^,\(\)]+)\s*\)/$1 + $2*'1 month'::interval/si; + + # Convert the call to the Oracle function add_years() into Pg syntax + $str =~ s/ADD_YEARS\s*\(([^,]+),\s*(\d+)\s*\)/$1 + '$2 year'::interval/si; + $str =~ s/ADD_YEARS\s*\(([^,]+),\s*([^,\(\)]+)\s*\)/$1 + $2*' year'::interval/si; + + # Translate numtodsinterval Oracle function + $str =~ s/(?:NUMTODSINTERVAL|NUMTOYMINTERVAL)\s*\(\s*([^,]+)\s*,\s*([^\)]+)\s*\)/($1 * ('1'||$2)::interval)/is; + + # REGEX_LIKE( string, pattern, flags ) + $str =~ s/REGEXP_LIKE\s*\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\)]+)\s*\)/"regexp_match($1, $2," . regex_flags($class, $3) . ") IS NOT NULL"/iges; + # REGEX_LIKE( string, pattern ) + $str =~ s/REGEXP_LIKE\s*\(\s*([^,]+)\s*,\s*([^\)]+)\s*\)/"regexp_match($1, $2," . regex_flags($class, '') . ") IS NOT NULL"/iges; + + # REGEX_COUNT( string, pattern, position, flags ) + $str =~ s/REGEXP_COUNT\s*\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*(\d+)\s*,\s*([^\)]+)\s*\)/"(SELECT count(*) FROM regexp_matches(substr($1, $3), $2, " . regex_flags($class, $4, 'g') . "))"/iges; + # REGEX_COUNT( string, pattern, position ) + $str =~ s/REGEXP_COUNT\s*\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*(\d+)\s*\)/(SELECT count(*) FROM regexp_matches(substr($1, $3), $2, 'g'))/igs; + # REGEX_COUNT( string, pattern ) + $str =~ s/REGEXP_COUNT\s*\(\s*([^,]+)\s*,\s*([^\)]+)\s*\)/(SELECT count(*) FROM regexp_matches($1, $2, 'g'))/igs; + # REGEX_SUBSTR( string, pattern, pos, num ) translation + $str =~ s/REGEXP_SUBSTR\s*\(\s*([^\)]+)\s*\)/convert_regex_substr($class, $1)/iges; + } + + # Replace INSTR by POSITION + $str =~ s/\bINSTR\s*\(\s*([^,]+),\s*([^\),]+)\s*\)/position($2 in $1)/is; + $str =~ s/\bINSTR\s*\(\s*([^,]+),\s*([^,]+)\s*,\s*1\s*\)/position($2 in $1)/is; + + # The to_number() function reclaim a second argument under postgres which is the format. + # Replace to_number with a cast when no specific format is given + if (lc($class->{to_number_conversion}) ne 'none') + { + if ($class->{to_number_conversion} =~ /(numeric|bigint|integer|int)/i) + { + my $cast = lc($1); + if ($class->{type} ne 'TABLE') { + $str =~ s/\bTO_NUMBER\s*\(\s*([^,\)]+)\s*\)\s?/($1)\:\:$cast /is; + } else { + $str =~ s/\bTO_NUMBER\s*\(\s*([^,\)]+)\s*\)\s?/($1\:\:$cast) /is; + } + } + else + { + $str =~ s/\bTO_NUMBER\s*\(\s*([^,\)]+)\s*\)/to_number\($1,'$class->{to_number_conversion}'\)/is; + } + } + + # Replace the UTC convertion with the PG syntaxe + $str =~ s/SYS_EXTRACT_UTC\s*\(([^\)]+)\)/($1 AT TIME ZONE 'UTC')/is; + + # Remove call to XMLCDATA, there's no such function with PostgreSQL + $str =~ s/XMLCDATA\s*\(([^\)]+)\)/''/is; + # Remove call to getClobVal() or getStringVal, no need of that + $str =~ s/\.(getClobVal|getStringVal)\s*\(\s*\)//is; + # Add the name keyword to XMLELEMENT + $str =~ s/XMLELEMENT\s*\(\s*/XMLELEMENT(name /is; + + # Cast round() call as numeric + $str =~ s/round\s*\(([^,]+),([\s\d]+)\)/round\(($1)::numeric,$2\)/is; + + if ($str =~ /SDO_/is) + { + # Replace SDO_GEOM to the postgis equivalent + $str = &replace_sdo_function($str); + + # Replace Spatial Operator to the postgis equivalent + $str = &replace_sdo_operator($str); + } + + # Rewrite replace(a,b) with three argument + $str =~ s/REPLACE\s*\($field,$field\)/replace($1, $2, '')/is; + + # Replace Oracle substr(string, start_position, length) with + # PostgreSQL substring(string from start_position for length) + $str =~ s/\bsubstrb\s*\(/substr\(/igs; + if (!$class->{pg_supports_substr}) + { + $str =~ s/\bsubstr\s*\($field,$field,$field\)/substring($1 from $2 for $3)/is; + $str =~ s/\bsubstr\s*\($field,$field\)/substring($1 from $2)/is; + } + + # Replace call to function with out parameters + $str = replace_out_param_call($class, $str); + + # Replace some sys_context call to the postgresql equivalent + if ($str =~ /SYS_CONTEXT/is) { + replace_sys_context($str); + } + + return $str; +} + +############## +# Replace call to function with out parameters +############## +sub replace_out_param_call +{ + my ($class, $str) = @_; + + if (uc($class->{type}) =~ /^(PACKAGE|FUNCTION|PROCEDURE|TRIGGER)$/) { + foreach my $sch (sort keys %{$class->{function_metadata}}) { + foreach my $p (sort keys %{$class->{function_metadata}{$sch}}) { + foreach my $k (sort keys %{$class->{function_metadata}{$sch}{$p}}) { + if ($class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout}) { + my $fct_name = $class->{function_metadata}{$sch}{$p}{$k}{metadata}{fct_name} || ''; + next if (!$fct_name); + next if ($p eq 'none' && $str !~ /\b$fct_name\b/is); + next if ($p ne 'none' && $str !~ /\b$p\.$fct_name\b/is && $str !~ /(^|[^\.])\b$fct_name\b/is); + + # Prevent replacement with same function name from an other package + next if ($class->{current_package} && lc($p) ne lc($class->{current_package}) && $str =~ /(^|[^\.])\b$fct_name\b/is); + + my %replace_out_parm = (); + my $idx = 0; + while ($str =~ s/((?:[^\s\.]+\.)?\b$fct_name)\s*\(([^\(\)]+)\)/\%FCTINOUTPARAM$idx\%/is) { + my $fname = $1; + my $fparam = $2; + if ($fname =~ /\./ && lc($fname) ne lc($k)) { + $replace_out_parm{$idx} = "$fname($fparam)"; + next; + } + $replace_out_parm{$idx} = "$fname("; + # Extract position of out parameters + my @params = split(/\s*,\s*/, $class->{function_metadata}{$sch}{$p}{$k}{metadata}{args}); + my @cparams = split(/\s*,\s*/, $fparam); + my $call_params = ''; + my @out_pos = (); + my @out_fields = (); + for (my $i = 0; $i <= $#params; $i++) { + if (!$class->{is_mysql} && $params[$i] =~ /\s*([^\s]+)\s+(OUT|INOUT)\s/is) { + push(@out_fields, $1); + push(@out_pos, $i); + $call_params .= $cparams[$i] if ($params[$i] =~ /\bINOUT\b/is); + } elsif ($class->{is_mysql} && $params[$i] =~ /\s*(OUT|INOUT)\s+([^\s]+)\s/is) { + push(@out_fields, $2); + push(@out_pos, $i); + $call_params .= $cparams[$i] if ($params[$i] =~ /\bINOUT\b/is); + } else { + $call_params .= $cparams[$i]; + } + $call_params .= ", " if ($i < $#params); + } + map { s/^\(//; } @out_fields; + $call_params =~ s/(\s*,\s*)+$//s; + while ($call_params =~ s/\s*,\s*,\s*/, /s) {}; + $call_params =~ s/^(\s*,\s*)+//s; + $replace_out_parm{$idx} .= "$call_params)"; + my @out_param = (); + foreach my $i (@out_pos) { + push(@out_param, $cparams[$i]); + } + if ($class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout} == 1) { + if ($#out_param == 0) { + $replace_out_parm{$idx} = "$out_param[0] := $replace_out_parm{$idx}"; + } else { + $replace_out_parm{$idx} = "SELECT * FROM $replace_out_parm{$idx} INTO " . join(', ', @out_param); + } + } elsif ($class->{function_metadata}{$sch}{$p}{$k}{metadata}{inout} > 1) { + $class->{replace_out_params} = "_ora2pg_r RECORD;" if (!$class->{replace_out_params}); + $replace_out_parm{$idx} = "SELECT * FROM $replace_out_parm{$idx} INTO _ora2pg_r;"; + my $out_field_pos = 0; + foreach $param (@out_param) { + $replace_out_parm{$idx} .= " $param := _ora2pg_r.$out_fields[$out_field_pos++];"; + } + $replace_out_parm{$idx} =~ s/;$//s; + } + $idx++; + } + $str =~ s/\%FCTINOUTPARAM(\d+)\%/$replace_out_parm{$1}/gs; + } + } + } + } + } + return $str; +} + +# Replace decode("user_status",'active',"username",null) +# PostgreSQL (CASE WHEN "user_status"='ACTIVE' THEN "username" ELSE NULL END) +sub replace_decode +{ + my $str = shift; + + while ($str =~ s/\bDECODE\s*\((.*)$/\%DECODE\%/is) { + my @decode_params = (''); + my $stop_learning = 0; + my $idx = 1; + foreach my $c (split(//, $1)) { + $idx++ if (!$stop_learning && $c eq '('); + $idx-- if (!$stop_learning && $c eq ')'); + + if ($idx == 0) { + # Do not copy last parenthesis in the output string + $c = '' if (!$stop_learning); + # Inform the loop that we don't want to process any charater anymore + $stop_learning = 1; + # We have reach the end of the decode() parameter + # next character must be restored to the final string. + $str .= $c; + } elsif ($idx > 0) { + # We are parsing the decode() parameter part, append + # the caracter to the right part of the param array. + if ($c eq ',' && ($idx - 1) == 0) { + # we are switching to a new parameter + push(@decode_params, ''); + } elsif ($c ne "\n") { + $decode_params[-1] .= $c; + } + } + } + my $case_str = 'CASE '; + for (my $i = 1; $i <= $#decode_params; $i+=2) { + $decode_params[$i] =~ s/^\s+//gs; + $decode_params[$i] =~ s/\s+$//gs; + if ($i < $#decode_params) { + $case_str .= "WHEN $decode_params[0]=$decode_params[$i] THEN $decode_params[$i+1] "; + } else { + $case_str .= " ELSE $decode_params[$i] "; + } + } + $case_str .= 'END '; + $str =~ s/\%DECODE\%/$case_str/s; + } + + return $str; +} + +# Function to replace call to SYS_CONTECT('USERENV', ...) +# Possibly corresponding PostgreSQL variables: http://www.postgresql.org/docs/current/static/functions-info.html +sub replace_sys_context +{ + my $str = shift; + + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(OS_USER|SESSION_USER|AUTHENTICATED_IDENTITY)'\s*\)/session_user/is; + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'BG_JOB_ID'\s*\)/pg_backend_pid()/is; + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(CLIENT_IDENTIFIER|PROXY_USER)'\s*\)/session_user/is; + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'CURRENT_SCHEMA'\s*\)/current_schema/is; + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'CURRENT_USER'\s*\)/current_user/is; + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(DB_NAME|DB_UNIQUE_NAME)'\s*\)/current_database/is; + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'(HOST|IP_ADDRESS)'\s*\)/inet_client_addr()/is; + $str =~ s/SYS_CONTEXT\s*\(\s*'USERENV'\s*,\s*'SERVER_HOST'\s*\)/inet_server_addr()/is; + + return $str; +} + +sub replace_sdo_function +{ + my $str = shift; + + $str =~ s/SDO_GEOM\.RELATE/ST_Relate/igs; + $str =~ s/SDO_GEOM\.VALIDATE_GEOMETRY_WITH_CONTEXT/ST_IsValidReason/igs; + $str =~ s/SDO_GEOM\.WITHIN_DISTANCE/ST_DWithin/igs; + $str =~ s/SDO_GEOM\.//igs; + $str =~ s/SDO_DISTANCE/ST_Distance/igs; + $str =~ s/SDO_BUFFER/ST_Buffer/igs; + $str =~ s/SDO_CENTROID/ST_Centroid/igs; + $str =~ s/SDO_UTIL\.GETVERTICES/ST_DumpPoints/igs; + $str =~ s/SDO_TRANSLATE/ST_Translate/igs; + $str =~ s/SDO_SIMPLIFY/ST_Simplify/igs; + $str =~ s/SDO_AREA/ST_Area/igs; + $str =~ s/SDO_CONVEXHULL/ST_ConvexHull/igs; + $str =~ s/SDO_DIFFERENCE/ST_Difference/igs; + $str =~ s/SDO_INTERSECTION/ST_Intersection/igs; + $str =~ s/SDO_LENGTH/ST_Length/igs; + $str =~ s/SDO_POINTONSURFACE/ST_PointOnSurface/igs; + $str =~ s/SDO_UNION/ST_Union/igs; + $str =~ s/SDO_XOR/ST_SymDifference/igs; + + # Note that with ST_DumpPoints and : + # TABLE(SDO_UTIL.GETVERTICES(C.GEOLOC)) T + # T.X, T.Y, T.ID must be replaced manually as ST_X(T.geom) X, ST_Y(T.geom) Y, (T).path[1] ID + my $field = '\s*[^\(\),]+\s*'; + my $num_field = '\s*[\d\.]+\s*'; + + # SDO_GEOM.RELATE(geom1 IN SDO_GEOMETRY,mask IN VARCHAR2,geom2 IN SDO_GEOMETRY,tol IN NUMBER) + $str =~ s/(ST_Relate\s*\($field),$field,($field),($field)\)/$1,$2\)/is; + # SDO_GEOM.RELATE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,mask IN VARCHAR2,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) + $str =~ s/(ST_Relate\s*\($field),$field,$field,($field),$field\)/$1,$2\)/is; + # SDO_GEOM.SDO_AREA(geom IN SDO_GEOMETRY, tol IN NUMBER [, unit IN VARCHAR2]) + # SDO_GEOM.SDO_AREA(geom IN SDO_GEOMETRY,dim IN SDO_DIM_ARRAY [, unit IN VARCHAR2]) + $str =~ s/(ST_Area\s*\($field),[^\)]+\)/$1\)/is; + # SDO_GEOM.SDO_BUFFER(geom IN SDO_GEOMETRY,dist IN NUMBER, tol IN NUMBER [, params IN VARCHAR2]) + $str =~ s/(ST_Buffer\s*\($field,$num_field),[^\)]+\)/$1\)/is; + # SDO_GEOM.SDO_BUFFER(geom IN SDO_GEOMETRY,dim IN SDO_DIM_ARRAY,dist IN NUMBER [, params IN VARCHAR2]) + $str =~ s/(ST_Buffer\s*\($field),$field,($num_field)[^\)]*\)/$1,$2\)/is; + # SDO_GEOM.SDO_CENTROID(geom1 IN SDO_GEOMETRY,tol IN NUMBER) + # SDO_GEOM.SDO_CENTROID(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY) + $str =~ s/(ST_Centroid\s*\($field),$field\)/$1\)/is; + # SDO_GEOM.SDO_CONVEXHULL(geom1 IN SDO_GEOMETRY,tol IN NUMBER) + # SDO_GEOM.SDO_CONVEXHULL(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY) + $str =~ s/(ST_ConvexHull\s*\($field),$field\)/$1\)/is; + # SDO_GEOM.SDO_DIFFERENCE(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY,tol IN NUMBER) + $str =~ s/(ST_Difference\s*\($field,$field),$field\)/$1\)/is; + # SDO_GEOM.SDO_DIFFERENCE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) + $str =~ s/(ST_Difference\s*\($field),$field,($field),$field\)/$1,$2\)/is; + # SDO_GEOM.SDO_DISTANCE(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY,tol IN NUMBER [, unit IN VARCHAR2]) + $str =~ s/(ST_Distance\s*\($field,$field),($num_field)[^\)]*\)/$1\)/is; + # SDO_GEOM.SDO_DISTANCE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY [, unit IN VARCHAR2]) + $str =~ s/(ST_Distance\s*\($field),$field,($field),($field)[^\)]*\)/$1,$2\)/is; + # SDO_GEOM.SDO_INTERSECTION(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY,tol IN NUMBER) + $str =~ s/(ST_Intersection\s*\($field,$field),$field\)/$1\)/is; + # SDO_GEOM.SDO_INTERSECTION(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) + $str =~ s/(ST_Intersection\s*\($field),$field,($field),$field\)/$1,$2\)/is; + # SDO_GEOM.SDO_LENGTH(geom IN SDO_GEOMETRY, dim IN SDO_DIM_ARRAY [, unit IN VARCHAR2]) + # SDO_GEOM.SDO_LENGTH(geom IN SDO_GEOMETRY, tol IN NUMBER [, unit IN VARCHAR2]) + $str =~ s/(ST_Length\s*\($field),($field)[^\)]*\)/$1\)/is; + # SDO_GEOM.SDO_POINTONSURFACE(geom1 IN SDO_GEOMETRY, tol IN NUMBER) + # SDO_GEOM.SDO_POINTONSURFACE(geom1 IN SDO_GEOMETRY, dim1 IN SDO_DIM_ARRAY) + $str =~ s/(ST_PointOnSurface\s*\($field),$field\)/$1\)/is; + # SDO_GEOM.SDO_UNION(geom1 IN SDO_GEOMETRY, geom2 IN SDO_GEOMETRY, tol IN NUMBER) + $str =~ s/(ST_Union\s*\($field,$field),$field\)/$1\)/is; + # SDO_GEOM.SDO_UNION(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) + $str =~ s/(ST_Union\s*\($field),$field,($field),$field\)/$1,$2\)/is; + # SDO_GEOM.SDO_XOR(geom1 IN SDO_GEOMETRY,geom2 IN SDO_GEOMETRY, tol IN NUMBER) + $str =~ s/(ST_SymDifference\s*\($field,$field),$field\)/$1\)/is; + # SDO_GEOM.SDO_XOR(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY) + $str =~ s/(ST_SymDifference\s*\($field),$field,($field),$field\)/$1,$2\)/is; + # SDO_GEOM.VALIDATE_GEOMETRY_WITH_CONTEXT(geom1 IN SDO_GEOMETRY, tol IN NUMBER) + # SDO_GEOM.VALIDATE_GEOMETRY_WITH_CONTEXT(geom1 IN SDO_GEOMETRY, dim1 IN SDO_DIM_ARRAY) + $str =~ s/(ST_IsValidReason\s*\($field),$field\)/$1\)/is; + # SDO_GEOM.WITHIN_DISTANCE(geom1 IN SDO_GEOMETRY,dim1 IN SDO_DIM_ARRAY,dist IN NUMBER,geom2 IN SDO_GEOMETRY,dim2 IN SDO_DIM_ARRAY [, units IN VARCHAR2]) + $str =~ s/(ST_DWithin\s*\($field),$field,($field),($field),($field)[^\)]*\)/$1,$3,$2\)/is; + # SDO_GEOM.WITHIN_DISTANCE(geom1 IN SDO_GEOMETRY,dist IN NUMBER,geom2 IN SDO_GEOMETRY, tol IN NUMBER [, units IN VARCHAR2]) + $str =~ s/(ST_DWithin\s*\($field)(,$field)(,$field),($field)[^\)]*\)/$1$3$2\)/is; + + return $str; +} + +sub replace_sdo_operator +{ + my $str = shift; + + # SDO_CONTAINS(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_CONTAINS\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Contains($1)/is; + $str =~ s/SDO_CONTAINS\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Contains($1)/is; + $str =~ s/SDO_CONTAINS\s*\(([^\)]+)\)/ST_Contains($1)/is; + # SDO_RELATE(geometry1, geometry2, param) = 'TRUE' + $str =~ s/SDO_RELATE\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Relate($1)/is; + $str =~ s/SDO_RELATE\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Relate($1)/is; + $str =~ s/SDO_RELATE\s*\(([^\)]+)\)/ST_Relate($1)/is; + # SDO_WITHIN_DISTANCE(geometry1, aGeom, params) = 'TRUE' + $str =~ s/SDO_WITHIN_DISTANCE\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_DWithin($1)/is; + $str =~ s/SDO_WITHIN_DISTANCE\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_DWithin($1)/is; + $str =~ s/SDO_WITHIN_DISTANCE\s*\(([^\)]+)\)/ST_DWithin($1)/is; + # SDO_TOUCH(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_TOUCH\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Touches($1)/is; + $str =~ s/SDO_TOUCH\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Touches($1)/is; + $str =~ s/SDO_TOUCH\s*\(([^\)]+)\)/ST_Touches($1)/is; + # SDO_OVERLAPS(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_OVERLAPS\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Overlaps($1)/is; + $str =~ s/SDO_OVERLAPS\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Overlaps($1)/is; + $str =~ s/SDO_OVERLAPS\s*\(([^\)]+)\)/ST_Overlaps($1)/is; + # SDO_INSIDE(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_INSIDE\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Within($1)/is; + $str =~ s/SDO_INSIDE\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Within($1)/is; + $str =~ s/SDO_INSIDE\s*\(([^\)]+)\)/ST_Within($1)/is; + # SDO_EQUAL(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_EQUAL\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Equals($1)/is; + $str =~ s/SDO_EQUAL\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Equals($1)/is; + $str =~ s/SDO_EQUAL\s*\(([^\)]+)\)/ST_Equals($1)/is; + # SDO_COVERS(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_COVERS\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Covers($1)/is; + $str =~ s/SDO_COVERS\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Covers($1)/is; + $str =~ s/SDO_COVERS\s*\(([^\)]+)\)/ST_Covers($1)/is; + # SDO_COVEREDBY(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_COVEREDBY\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_CoveredBy($1)/is; + $str =~ s/SDO_COVEREDBY\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_CoveredBy($1)/is; + $str =~ s/SDO_COVEREDBY\s*\(([^\)]+)\)/ST_CoveredBy($1)/is; + # SDO_ANYINTERACT(geometry1, geometry2) = 'TRUE' + $str =~ s/SDO_ANYINTERACT\s*\((.*?)\)\s*=\s*[']+TRUE[']+/ST_Intersects($1)/is; + $str =~ s/SDO_ANYINTERACT\s*\((.*?)\)\s*=\s*[']+FALSE[']+/NOT ST_Intersects($1)/is; + $str =~ s/SDO_ANYINTERACT\s*\(([^\)]+)\)/ST_Intersects($1)/is; + + return $str; +} + +# Function used to rewrite dbms_output.put, dbms_output.put_line and +# dbms_output.new_line by a plpgsql code +sub raise_output +{ + my ($class, $str) = @_; + + my @strings = split(/\s*\|\|\s*/s, $str); + + my @params = (); + my @pattern = (); + foreach my $el (@strings) { + $el =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/gs; + $el =~ s/ORA2PG_ESCAPE2_QUOTE/''/gs; + $el =~ s/ORA2PG_ESCAPE1_QUOTE'/\\'/gs; + if ($el =~ /^\s*'(.*)'\s*$/s) { + push(@pattern, $1); + } else { + push(@pattern, '%'); + push(@params, $el); + } + } + #my $ret = "RAISE NOTICE '$pattern'"; + my $ret = "'" . join('', @pattern) . "'"; + $ret =~ s/\%\%/\% \%/gs; + if ($#params >= 0) { + $ret .= ', ' . join(', ', @params); + } + + return 'RAISE NOTICE ' . $ret; +} + +sub replace_sql_type +{ + my ($str, $pg_numeric_type, $default_numeric, $pg_integer_type, %data_type) = @_; + + + $str =~ s/with local time zone/with time zone/igs; + $str =~ s/([A-Z])\%ORA2PG_COMMENT/$1 \%ORA2PG_COMMENT/igs; + + # Replace MySQL type UNSIGNED in cast + $str =~ s/UNSIGNED\s*\)/bigint)/is; + + # Remove precision for RAW|BLOB as type modifier is not allowed for type "bytea" + $str =~ s/\b(RAW|BLOB)\s*\(\s*\d+\s*\)/$1/igs; + + # Replace type with precision + my @ora_type = keys %data_type; + map { s/\(/\\\(/; s/\)/\\\)/; } @ora_type; + my $oratype_regex = join('|', @ora_type); + + while ($str =~ /(.*)\b($oratype_regex)\s*\(([^\)]+)\)/i) + { + my $backstr = $1; + my $type = uc($2); + my $args = $3; + + # Remove extra CHAR or BYTE information from column type + $args =~ s/\s*(CHAR|BYTE)\s*$//i; + if ($backstr =~ /_$/) + { + $str =~ s/\b($oratype_regex)\s*\(([^\)]+)\)/$1\%\|$2\%\|\%/is; + next; + } + + my ($precision, $scale) = split(/\s*,\s*/, $args); + $precision = '' if ($precision eq '*'); # case of NUMBER(*,10) + $scale ||= 0; + my $len = $precision || 0; + $len =~ s/\D//; + if ( $type =~ /CHAR|STRING/i ) + { + # Type CHAR have default length set to 1 + # Type VARCHAR(2) must have a specified length + $len = 1 if (!$len && (($type eq "CHAR") || ($type eq "NCHAR"))); + $str =~ s/\b$type\b\s*\([^\)]+\)/$data_type{$type}\%\|$len\%\|\%/is; + } + elsif ($type =~ /TIMESTAMP/i) + { + $len = 6 if ($len > 6); + $str =~ s/\b$type\b\s*\([^\)]+\)/timestamp\%\|$len%\|\%/is; + } + elsif ($type =~ /INTERVAL/i) + { + # Interval precision for year/month/day is not supported by PostgreSQL + $str =~ s/(INTERVAL\s+YEAR)\s*\(\d+\)/$1/is; + $str =~ s/(INTERVAL\s+YEAR\s+TO\s+MONTH)\s*\(\d+\)/$1/is; + $str =~ s/(INTERVAL\s+DAY)\s*\(\d+\)/$1/is; + # maximum precision allowed for seconds is 6 + if ($str =~ /INTERVAL\s+DAY\s+TO\s+SECOND\s*\((\d+)\)/) + { + if ($1 > 6) { + $str =~ s/(INTERVAL\s+DAY\s+TO\s+SECOND)\s*\(\d+\)/$1(6)/i; + } + } + } + elsif ($type eq "NUMBER") + { + # This is an integer + if (!$scale) + { + if ($precision) + { + if ($pg_integer_type) + { + if ($precision < 5) { + $str =~ s/\b$type\b\s*\([^\)]+\)/smallint/is; + } elsif ($precision <= 9) { + $str =~ s/\b$type\b\s*\([^\)]+\)/integer/is; + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/bigint/is; + } + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/numeric\%\|$precision\%\|\%/i; + } + } + elsif ($pg_integer_type) + { + my $tmp = $default_numeric || 'bigint'; + $str =~ s/\b$type\b\s*\([^\)]+\)/$tmp/is; + } + } + else + { + if ($pg_numeric_type) + { + if ($precision eq '') { + $str =~ s/\b$type\b\s*\([^\)]+\)/decimal(38, $scale)/is; + } elsif ($precision <= 6) { + $str =~ s/\b$type\b\s*\([^\)]+\)/real/is; + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/double precision/is; + } + } + else + { + if ($precision eq '') { + $str =~ s/\b$type\b\s*\([^\)]+\)/decimal(38, $scale)/is; + } else { + $str =~ s/\b$type\b\s*\([^\)]+\)/decimal\%\|$precision,$scale\%\|\%/is; + } + } + } + } + elsif ($type eq "NUMERIC") { + $str =~ s/\b$type\b\s*\([^\)]+\)/numeric\%\|$args\%\|\%/is; + } elsif ( ($type eq "DEC") || ($type eq "DECIMAL") ) { + $str =~ s/\b$type\b\s*\([^\)]+\)/decimal\%\|$args\%\|\%/is; + } + else + { + # Prevent from infinit loop + $str =~ s/\(/\%\|/s; + $str =~ s/\)/\%\|\%/s; + } + } + $str =~ s/\%\|\%/\)/gs; + $str =~ s/\%\|/\(/gs; + + # Replace datatype without precision + my $number = $data_type{'NUMBER'}; + $number = $default_numeric if ($pg_integer_type); + $str =~ s/\bNUMBER\b/$number/igs; + + # Set varchar without length to text + $str =~ s/\bVARCHAR2\b/VARCHAR/igs; + $str =~ s/\bSTRING\b/VARCHAR/igs; + $str =~ s/\bVARCHAR(\s*(?!\())/text$1/igs; + + foreach my $t ('DATE','LONG RAW','LONG','NCLOB','CLOB','BLOB','BFILE','RAW','ROWID','UROWID','FLOAT','DOUBLE PRECISION','INTEGER','INT','REAL','SMALLINT','BINARY_FLOAT','BINARY_DOUBLE','BINARY_INTEGER','BOOLEAN','XMLTYPE','SDO_GEOMETRY','PLS_INTEGER') { + $str =~ s/\b$t\b/$data_type{$t}/igs; + } + + # Translate cursor declaration + $str = replace_cursor_def($str); + + # Remove remaining %ROWTYPE in other prototype declaration + #$str =~ s/\%ROWTYPE//isg; + + $str =~ s/;[ ]+/;/gs; + + return $str; +} + +sub replace_cursor_def +{ + my $str = shift; + + # Remove IN information from cursor declaration + while ($str =~ s/(\bCURSOR\b[^\(]+)\(([^\)]+\bIN\b[^\)]+)\)/$1\(\%\%CURSORREPLACE\%\%\)/is) { + my $args = $2; + $args =~ s/\bIN\b//igs; + $str =~ s/\%\%CURSORREPLACE\%\%/$args/is; + } + + # Replace %ROWTYPE ref cursor + $str =~ s/\bTYPE\s+([^\s]+)\s+(IS\s+REF\s+CURSOR|REFCURSOR)\s+RETURN\s+[^\s\%]+\%ROWTYPE;/$1 REFCURSOR;/isg; + + + # Replace local type ref cursor + my %locatype = (); + my $i = 0; + while ($str =~ s/\bTYPE\s+([^\s]+)\s+(IS\s+REF\s+CURSOR|REFCURSOR)\s*;/\%LOCALTYPE$i\%/is) { + $localtype{$i} = "TYPE $1 IS REF CURSOR;"; + my $local_type = $1; + if ($str =~ s/\b([^\s]+)\s+$local_type\s*;/$1 REFCURSOR;/igs) { + $str =~ s/\%LOCALTYPE$i\%//igs; + } + $i++; + } + $str =~ s/\%LOCALTYPE(\d+)\%/$localtype{$1}/gs; + + # Retrieve cursor names + #my @cursor_names = $str =~ /\bCURSOR\b\s*([A-Z0-9_\$]+)/isg; + # Reorder cursor declaration + $str =~ s/\bCURSOR\b\s*([A-Z0-9_\$]+)/$1 CURSOR/isg; + + # Replace call to cursor type if any + #foreach my $c (@cursor_names) { + # $str =~ s/\b$c\%ROWTYPE/RECORD/isg; + #} + + # Replace REF CURSOR as Pg REFCURSOR + $str =~ s/\bIS(\s*)REF\s+CURSOR/REFCURSOR/isg; + $str =~ s/\bREF\s+CURSOR/REFCURSOR/isg; + + # Replace SYS_REFCURSOR as Pg REFCURSOR + $str =~ s/\bSYS_REFCURSOR\b/REFCURSOR/isg; + + # Replace CURSOR IS SELECT by CURSOR FOR SELECT + $str =~ s/\bCURSOR(\s+)IS(\s+)(\%ORA2PG_COMMENT\d+\%)?(\s*)SELECT/CURSOR$1FOR$2$3$4SELECT/isg; + # Replace CURSOR (param) IS SELECT by CURSOR FOR SELECT + $str =~ s/\bCURSOR(\s*\([^\)]+\)\s*)IS(\s*)(\%ORA2PG_COMMENT\d+\%)?(\s*)SELECT/CURSOR$1FOR$2$3$4SELECT/isg; + + # Replace REF CURSOR as Pg REFCURSOR + $str =~ s/\bIS(\s*)REF\s+CURSOR/REFCURSOR/isg; + $str =~ s/\bREF\s+CURSOR/REFCURSOR/isg; + + # Replace SYS_REFCURSOR as Pg REFCURSOR + $str =~ s/\bSYS_REFCURSOR\b/REFCURSOR/isg; + + # Replace OPEN cursor FOR with dynamic query + $str =~ s/(OPEN\s+(?:[^;]+?)\s+FOR)((?:[^;]+?)USING)/$1 EXECUTE$2/isg; + $str =~ s/(OPEN\s+(?:[^;]+?)\s+FOR)\s+((?!EXECUTE)(?:[^;]+?)\|\|)/$1 EXECUTE $2/isg; + $str =~ s/(OPEN\s+(?:[^;]+?)\s+FOR)\s+([^\s]+\s*;)/$1 EXECUTE $2/isg; + # Remove empty parenthesis after an open cursor + $str =~ s/(OPEN\s+[^\(\s;]+)\s*\(\s*\)/$1/isg; + + # Invert FOR CURSOR call + $str =~ s/\bFOR\s+CURSOR(\s+)/CURSOR FOR$1/igs; + + return $str; +} + +sub estimate_cost +{ + my ($class, $str, $type) = @_; + + return mysql_estimate_cost($str, $type) if ($class->{is_mysql}); + + my %cost_details = (); + + # Remove some unused pragma from the cost assessment + $str =~ s/PRAGMA RESTRICT_REFERENCES[^;]+;//igs; + $str =~ s/PRAGMA SERIALLY_REUSABLE[^;]*;//igs; + $str =~ s/PRAGMA INLINE[^;]+;//igs; + + # Default cost is testing that mean it at least must be tested + my $cost = $FCT_TEST_SCORE; + # When evaluating queries size must not be included here + if ($type eq 'QUERY' || $type eq 'VIEW') { + $cost = 0; + } + $cost_details{'TEST'} = $cost; + + # Set cost following code length + my $cost_size = int(length($str)/$SIZE_SCORE) || 1; + # When evaluating queries size must not be included here + if ($type eq 'QUERY' || $type eq 'VIEW') { + $cost_size = 0; + } + $cost += $cost_size; + $cost_details{'SIZE'} = $cost_size; + + # Try to figure out the manual work + my $n = () = $str =~ m/\bTRUNC\s*\(/igs; + $cost_details{'TRUNC'} += $n; + $n = () = $str =~ m/\bIS\s+TABLE\s+OF\b/igs; + $cost_details{'IS TABLE OF'} += $n; + $n = () = $str =~ m/\(\+\)/igs; + $cost_details{'OUTER JOIN'} += $n; + $n = () = $str =~ m/\bCONNECT\s+BY\b/igs; + $cost_details{'CONNECT BY'} += $n; + $n = () = $str =~ m/\bBULK\s+COLLECT\b/igs; + $cost_details{'BULK COLLECT'} += $n; + $n = () = $str =~ m/\bFORALL\b/igs; + $cost_details{'FORALL'} += $n; + $n = () = $str =~ m/\bGOTO\b/igs; + $cost_details{'GOTO'} += $n; + $n = () = $str =~ m/\bROWNUM\b/igs; + $cost_details{'ROWNUM'} += $n; + $n = () = $str =~ m/\bNOTFOUND\b/igs; + $cost_details{'NOTFOUND'} += $n; + $n = () = $str =~ m/\bROWID\b/igs; + $cost_details{'ROWID'} += $n; + $n = () = $str =~ m/\bUROWID\b/igs; + $cost_details{'UROWID'} += $n; + $n = () = $str =~ m/\bSQLSTATE\b/igs; + $cost_details{'SQLCODE'} += $n; + $n = () = $str =~ m/\bIS RECORD\b/igs; + $cost_details{'IS RECORD'} += $n; + $n = () = $str =~ m/FROM[^;]*\bTABLE\s*\(/igs; + $cost_details{'TABLE'} += $n; + $n = () = $str =~ m/PIPE\s+ROW/igs; + $cost_details{'PIPE ROW'} += $n; + $n = () = $str =~ m/DBMS_\w/igs; + $cost_details{'DBMS_'} += $n; + $n = () = $str =~ m/DBMS_OUTPUT\.(put_line|new_line|put)/igs; + $cost_details{'DBMS_'} -= $n; + $n = () = $str =~ m/DBMS_OUTPUT\.put\(/igs; + $cost_details{'DBMS_OUTPUT.put'} += $n; + $n = () = $str =~ m/DBMS_STANDARD\.RAISE EXCEPTION/igs; + $cost_details{'DBMS_'} -= $n; + $n = () = $str =~ m/UTL_\w/igs; + $cost_details{'UTL_'} += $n; + $n = () = $str =~ m/CTX_\w/igs; + $cost_details{'CTX_'} += $n; + $n = () = $str =~ m/\bEXTRACT\s*\(/igs; + $cost_details{'EXTRACT'} += $n; + $n = () = $str =~ m/\bTO_NUMBER\s*\(/igs; + $cost_details{'TO_NUMBER'} += $n; + # See: http://www.postgresql.org/docs/9.0/static/errcodes-appendix.html#ERRCODES-TABLE + $n = () = $str =~ m/\b(DUP_VAL_ON_INDEX|TIMEOUT_ON_RESOURCE|TRANSACTION_BACKED_OUT|NOT_LOGGED_ON|LOGIN_DENIED|INVALID_NUMBER|PROGRAM_ERROR|VALUE_ERROR|ROWTYPE_MISMATCH|CURSOR_ALREADY_OPEN|ACCESS_INTO_NULL|COLLECTION_IS_NULL)\b/igs; + $cost_details{'EXCEPTION'} += $n; + if (!$class->{use_orafce}) + { + $n = () = $str =~ m/REGEXP_LIKE/igs; + $cost_details{'REGEXP_LIKE'} += $n; + $n = () = $str =~ m/REGEXP_SUBSTR/igs; + $cost_details{'REGEXP_SUBSTR'} += $n; + $n = () = $str =~ m/REGEXP_COUNT/igs; + $cost_details{'REGEXP_COUNT'} += $n; + $n = () = $str =~ m/REGEXP_INSTR/igs; + $cost_details{'REGEXP_INSTR'} += $n; + } + $n = () = $str =~ m/\b(INSERTING|DELETING|UPDATING)\b/igs; + $cost_details{'TG_OP'} += $n; + $n = () = $str =~ m/REF\s*CURSOR/igs; + $cost_details{'CURSOR'} += $n; + $n = () = $str =~ m/ORA_ROWSCN/igs; + $cost_details{'ORA_ROWSCN'} += $n; + $n = () = $str =~ m/SAVEPOINT/igs; + $cost_details{'SAVEPOINT'} += $n; + $n = () = $str =~ m/(FROM|EXEC)((?!WHERE).)*\b[\w\_]+\@[\w\_]+\b/igs; + $cost_details{'DBLINK'} += $n; + $n = () = $str =~ m/\%ISOPEN\b/igs; + $cost_details{'ISOPEN'} += $n; + $n = () = $str =~ m/\%ROWCOUNT\b/igs; + $cost_details{'ROWCOUNT'} += $n; + + $n = () = $str =~ m/PLVDATE/igs; + $cost_details{'PLVDATE'} += $n; + $n = () = $str =~ m/PLVSTR/igs; + $cost_details{'PLVSTR'} += $n; + $n = () = $str =~ m/PLVCHR/igs; + $cost_details{'PLVCHR'} += $n; + $n = () = $str =~ m/PLVSUBST/igs; + $cost_details{'PLVSUBST'} += $n; + $n = () = $str =~ m/PLVLEX/igs; + $cost_details{'PLVLEX'} += $n; + $n = () = $str =~ m/PLUNIT/igs; + $cost_details{'PLUNIT'} += $n; + $n = () = $str =~ m/ADD_MONTHS/igs; + $cost_details{'ADD_MONTHS'} += $n; + $n = () = $str =~ m/LAST_DAY/igs; + $cost_details{'LAST_DAY'} += $n; + $n = () = $str =~ m/NEXT_DAY/igs; + $cost_details{'NEXT_DAY'} += $n; + $n = () = $str =~ m/MONTHS_BETWEEN/igs; + $cost_details{'MONTHS_BETWEEN'} += $n; + $n = () = $str =~ m/NVL2/igs; + $cost_details{'NVL2'} += $n; + $str =~ s/MDSYS\.(["]*SDO_)/$1/igs; + $n = () = $str =~ m/SDO_\w/igs; + $cost_details{'SDO_'} += $n; + $n = () = $str =~ m/PRAGMA/igs; + $cost_details{'PRAGMA'} += $n; + $n = () = $str =~ m/MDSYS\./igs; + $cost_details{'MDSYS'} += $n; + $n = () = $str =~ m/MERGE\sINTO/igs; + $cost_details{'MERGE'} += $n; + $n = () = $str =~ m/\bCONTAINS\(/igs; + $cost_details{'CONTAINS'} += $n; + $n = () = $str =~ m/\bSCORE\((?:.*)?\bCONTAINS\(/igs; + $cost_details{'SCORE'} += $n; + $n = () = $str =~ m/CONTAINS\((?:.*)?\bFUZZY\(/igs; + $cost_details{'FUZZY'} += $n; + $n = () = $str =~ m/CONTAINS\((?:.*)?\bNEAR\(/igs; + $cost_details{'NEAR'} += $n; + $n = () = $str =~ m/TO_CHAR\([^,\)]+\)/igs; + $cost_details{'TO_CHAR'} += $n; + $n = () = $str =~ m/TO_NCHAR\([^,\)]+\)/igs; + $cost_details{'TO_NCHAR'} += $n; + $n = () = $str =~ m/\s+ANYDATA/igs; + $cost_details{'ANYDATA'} += $n; + $n = () = $str =~ m/\|\|/igs; + $cost_details{'CONCAT'} += $n; + $n = () = $str =~ m/TIMEZONE_(REGION|ABBR)/igs; + $cost_details{'TIMEZONE'} += $n; + $n = () = $str =~ m/IS\s+(NOT)?\s*JSON/igs; + $cost_details{'JSON'} += $n; + $n = () = $str =~ m/TO_CLOB\([^,\)]+\)/igs; + $cost_details{'TO_CLOB'} += $n; + + foreach my $f (@ORA_FUNCTIONS) { + if ($str =~ /\b$f\b/igs) { + $cost += 1; + $cost_details{$f} += 1; + } + } + foreach my $t (keys %UNCOVERED_SCORE) { + $cost += $UNCOVERED_SCORE{$t}*$cost_details{$t}; + } + + return $cost, %cost_details; +} + +=head2 mysql_to_plpgsql + +This function turn a MySQL function code into a PLPGSQL code + +=cut + +sub mysql_to_plpgsql +{ + my ($class, $str) = @_; + + # remove FROM DUAL + $str =~ s/FROM\s+DUAL//igs; + + # Simply remove this as not supported + $str =~ s/\bDEFAULT\s+NULL\b//igs; + + # Change mysql variable affectation + $str =~ s/\bSET\s+([^\s:=]+\s*)=([^;\n]+;)/$1:=$2/igs; + + # remove declared handler + $str =~ s/[^\s]+\s+HANDLER\s+FOR\s+[^;]+;//igs; + + # Fix call to unsigned + $str =~ s/UNSIGNED\sINTEGER/bigint/g; + $str =~ s/UNSIGNED\sINT/bigint/g; + $str =~ s/UNSIGNED/bigint/g; + + # Drop temporary doesn't exist in PostgreSQL + $str =~ s/DROP\s+TEMPORARY/DROP/gs; + + # Private temporary table doesn't exist in PostgreSQL + $str =~ s/PRIVATE\s+TEMPORARY/TEMPORARY/igs; + $str =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION/ON COMMIT PRESERVE ROWS/igs; + $str =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION/ON COMMIT DROP/igs; + + # Remove extra parenthesis in join in some possible cases + # ... INNER JOIN(services s) ON ... + $str =~ s/\bJOIN\s*\(([^\s]+\s+[^\s]+)\)/JOIN $1/igs; + + # Rewrite MySQL JOIN with WHERE clause instead of ON + $str =~ s/\((\s*[^\s]+(?:\s+[^\s]+)?\s+JOIN\s+[^\s]+(?:\s+[^\s]+)?\s*)\)\s+WHERE\s+/$1 ON /igs; + + # Try to replace LEAVE label by EXIT label + my %repl_leave = (); + my $i = 0; + while ($str =~ s/\bLEAVE\s+([^\s;]+)\s*;/%REPEXITLBL$i%/igs) { + my $label = $1; + if ( $str =~ /\b$label:/is) { + $repl_leave{$i} = "EXIT $label;"; + } else { + # This is a main block label + $repl_leave{$i} = "RETURN;"; + } + } + foreach $i (keys %repl_leave) { + $str =~ s/\%REPEXITLBL$i\%/$repl_leave{$i}/gs; + } + %repl_leave = (); + $str =~ s/\bLEAVE\s*;/EXIT;/igs; + + # Try to replace ITERATE label by CONTINUE label + my %repl_iterate = (); + $i = 0; + while ($str =~ s/\bITERATE\s+([^\s;]+)\s*;/%REPITERLBL$i%/igs) { + my $label = $1; + $repl_iterate{$i} = "CONTINUE $label;"; + } + foreach $i (keys %repl_iterate) { + $str =~ s/\%REPITERLBL$i\%/$repl_iterate{$i}/gs; + } + %repl_iterate = (); + $str =~ s/\bITERATE\s*;/CONTINUE;/igs; + + # Replace now() with CURRENT_TIMESTAMP even if this is the same + # because parenthesis can break the following regular expressions + $str =~ s/\bNOW\(\s*\)/CURRENT_TIMESTAMP/igs; + # Replace call to CURRENT_TIMESTAMP() to special variable + $str =~ s/\bCURRENT_TIMESTAMP\s*\(\)/CURRENT_TIMESTAMP/igs; + + # Replace EXTRACT() with unit not supported by PostgreSQL + if ($class->{mysql_internal_extract_format}) { + $str =~ s/\bEXTRACT\(\s*YEAR_MONTH\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'YYYYMM')::integer/igs; + $str =~ s/\bEXTRACT\(\s*DAY_HOUR\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24')::integer/igs; + $str =~ s/\bEXTRACT\(\s*DAY_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24MI')::integer/igs; + $str =~ s/\bEXTRACT\(\s*DAY_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24MISS')::integer/igs; + $str =~ s/\bEXTRACT\(\s*DAY_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DDHH24MISSUS')::bigint/igs; + $str =~ s/\bEXTRACT\(\s*HOUR_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24MI')::integer/igs; + $str =~ s/\bEXTRACT\(\s*HOUR_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24MISS')::integer/igs; + $str =~ s/\bEXTRACT\(\s*HOUR_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24MISSUS')::bigint/igs; + $str =~ s/\bEXTRACT\(\s*MINUTE_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MISS')::integer/igs; + $str =~ s/\bEXTRACT\(\s*MINUTE_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MISSUS')::bigint/igs; + $str =~ s/\bEXTRACT\(\s*SECOND_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'SSUS')::integer/igs; + } else { + $str =~ s/\bEXTRACT\(\s*YEAR_MONTH\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'YYYY-MM')/igs; + $str =~ s/\bEXTRACT\(\s*DAY_HOUR\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24')/igs; + $str =~ s/\bEXTRACT\(\s*DAY_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24:MI')/igs; + $str =~ s/\bEXTRACT\(\s*DAY_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24:MI:SS')/igs; + $str =~ s/\bEXTRACT\(\s*DAY_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'DD HH24:MI:SS.US')/igs; + $str =~ s/\bEXTRACT\(\s*HOUR_MINUTE\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24:MI')/igs; + $str =~ s/\bEXTRACT\(\s*HOUR_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24:MI:SS')/igs; + $str =~ s/\bEXTRACT\(\s*HOUR_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'HH24:MI:SS.US')/igs; + $str =~ s/\bEXTRACT\(\s*MINUTE_SECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MI:SS')/igs; + $str =~ s/\bEXTRACT\(\s*MINUTE_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'MI:SS.US')/igs; + $str =~ s/\bEXTRACT\(\s*SECOND_MICROSECOND\s+FROM\s+([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'SS.US')/igs; + } + + # Replace operators + if (!$class->{mysql_pipes_as_concat}) { + $str =~ s/\|\|/ OR /igs; + $str =~ s/\&\&/ AND /igs; + } + $str =~ s/BIT_XOR\(\s*([^,]+)\s*,\s*(\d+)\s*\)/$1 # coalesce($2, 0)/igs; + $str =~ s/\bXOR\b/#/igs; + $str =~ s/\b\^\b/#/igs; + + #### + # Replace some function with their PostgreSQL syntax + #### + + # Math related fucntion + $str =~ s/\bATAN\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/atan2($1, $2)/igs; + $str =~ s/\bLOG\(/ln\(/igs; + $str =~ s/\bLOG10\(\s*([^\(\)]+)\s*\)/log\(10, $1\)/igs; + $str =~ s/\bLOG2\(\s*([^\(\)]+)\s*\)/log\(2, $1\)/igs; + $str =~ s/([^\s]+)\s+MOD\s+([^\s]+)/mod\($1, $2\)/igs; + $str =~ s/\bPOW\(/power\(/igs; + $str =~ s/\bRAND\(\s*\)/random\(\)/igs; + + # Misc function + $str =~ s/\bCHARSET\(\s*([^\(\)]+)\s*\)/current_setting('server_encoding')/igs; + $str =~ s/\bCOLLATION\(\s*([^\(\)]+)\s*\)/current_setting('lc_collate')/igs; + $str =~ s/\bCONNECTION_ID\(\s*\)/pg_backend_pid()/igs; + $str =~ s/\b(DATABASE|SCHEMA)\(\s*\)/current_database()/igs; + $str =~ s/\bSLEEP\(/pg_sleep\(/igs; + $str =~ s/\bSYSTEM_USER\(\s*\)/CURRENT_USER/igs; + $str =~ s/\bSESSION_USER\(\s*\)/SESSION_USER/igs; + $str =~ s/\bTRUNCATE\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/trunc\($1, $2\)/igs; + $str =~ s/\bUSER\(\s*\)/CURRENT_USER/igs; + + # Date/time related function + $str =~ s/\b(CURDATE|CURRENT_DATE)\(\s*\)/CURRENT_DATE/igs; + $str =~ s/\b(CURTIME|CURRENT_TIME)\(\s*\)/LOCALTIME(0)/igs; + $str =~ s/\bCURRENT_TIMESTAMP\(\s*\)/CURRENT_TIMESTAMP::timestamp(0) without time zone/igs; + $str =~ s/\b(LOCALTIMESTAMP|LOCALTIME)\(\s*\)/CURRENT_TIMESTAMP::timestamp(0) without time zone/igs; + $str =~ s/\b(LOCALTIMESTAMP|LOCALTIME)\b/CURRENT_TIMESTAMP::timestamp(0) without time zone/igs; + $str =~ s/\bSYSDATE\(\s*\)/timeofday()::timestamp(0) without time zone/igs; + $str =~ s/\bUNIX_TIMESTAMP\(\s*\)/floor(extract(epoch from CURRENT_TIMESTAMP::timestamp with time zone))/igs; + $str =~ s/\bUNIX_TIMESTAMP\(\s*([^\)]+)\s*\)/floor(extract(epoch from ($1)::timestamp with time zone))/igs; + $str =~ s/\bUTC_DATE\(\s*\)/(CURRENT_TIMESTAMP AT TIME ZONE 'UTC')::date/igs; + $str =~ s/\bUTC_TIME\(\s*\)/(CURRENT_TIMESTAMP AT TIME ZONE 'UTC')::time(0)/igs; + $str =~ s/\bUTC_TIMESTAMP\(\s*\)/(CURRENT_TIMESTAMP AT TIME ZONE 'UTC')::timestamp(0)/igs; + + $str =~ s/\bCONVERT_TZ\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/(($1)::timestamp without time zone AT TIME ZONE ($2)::text) AT TIME ZONE ($3)::text/igs; + $str =~ s/\bDATEDIFF\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/extract(day from (date_trunc('day', ($1)::timestamp) - date_trunc('day', ($2)::timestamp)))/igs; + $str =~ s/\bDATE_FORMAT\(\s*(.*?)\s*,\s*('[^'\(\)]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_dateformat_to_pgsql($class, $1, $2)/iges; + $str =~ s/\b(?:ADDDATE|DATE_ADD)\(\s*(.*?)\s*,\s*INTERVAL\s*([^\(\),]+)\s*\)/"($1)::timestamp " . _replace_dateadd($2)/iges; + $str =~ s/\bADDDATE\(\s*([^,]+)\s*,\s*(\d+)\s*\)/($1)::timestamp + ($2 * interval '1 day')/igs; + $str =~ s/\bADDTIME\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp + ($2)::interval/igs; + + + $str =~ s/\b(DAY|DAYOFMONTH)\(\s*([^\(\)]+)\s*\)/extract(day from date($1))::integer/igs; + $str =~ s/\bDAYNAME\(\s*([^\(\)]+)\s*\)/to_char(($1)::date, 'FMDay')/igs; + $str =~ s/\bDAYOFWEEK\(\s*([^\(\)]+)\s*\)/extract(dow from date($1))::integer + 1/igs; # start on sunday = 1 + $str =~ s/\bDAYOFYEAR\(\s*([^\(\)]+)\s*\)/extract(doy from date($1))::integer/igs; + $str =~ s/\bFORMAT\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/to_char(round($1, $2), 'FM999,999,999,999,999,999,990'||case when $2 > 0 then '.'||repeat('0', $2) else '' end)/igs; + $str =~ s/\bFROM_DAYS\(\s*([^\(\)]+)\s*\)/'0001-01-01bc'::date + ($1)::integer/igs; + $str =~ s/\bFROM_UNIXTIME\(\s*([^\(\),]+)\s*\)/to_timestamp($1)::timestamp without time zone/igs; + $str =~ s/\bFROM_UNIXTIME\(\s*(.*?)\s*,\s*('[^\(\)]+'|\?TEXTVALUE\d+\?)\s*\)/FROM_UNIXTIME2(to_timestamp($1), $2)/igs; + $str =~ s/\bFROM_UNIXTIME2\(\s*(.*?)\s*,\s*('[^'\(\)]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_dateformat_to_pgsql($class, $1, $2)/eigs; + $str =~ s/\bGET_FORMAT\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/_mysql_getformat_to_pgsql($1, $2)/eigs; + $str =~ s/\bHOUR\(\s*([^\(\)]+)\s*\)/extract(hour from ($1)::interval)::integer/igs; + $str =~ s/\bLAST_DAY\(\s*([^\(\)]+)\s*\)/(date_trunc('month',($1)::timestamp + interval '1 month'))::date - 1/igs; + $str =~ s/\bMAKEDATE\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(date($1||'-01-01') + ($2 - 1) * interval '1 day')::date/igs; + $str =~ s/\bMAKETIME\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1 * interval '1 hour' + $2 * interval '1 min' + $3 * interval '1 sec')/igs; + $str =~ s/\bMICROSECOND\(\s*([^\(\)]+)\s*\)/extract(microsecond from ($1)::time)::integer/igs; + $str =~ s/\bMINUTE\(\s*([^\(\)]+)\s*\)/extract(minute from ($1)::time)::integer/igs; + $str =~ s/\bMONTH\(\s*([^\(\)]+)\s*\)/extract(month from date($1))::integer/igs; + $str =~ s/\bMONTHNAME\(\s*([^\(\)]+)\s*\)/to_char(($1)::date, 'FMMonth')/igs; + $str =~ s/\bQUARTER\(\s*([^\(\)]+)\s*\)/extract(quarter from date($1))::integer/igs; + $str =~ s/\bSECOND\(\s*([^\(\)]+)\s*\)/extract(second from ($1)::interval)::integer/igs; + $str =~ s/\bSEC_TO_TIME\(\s*([^\(\)]+)\s*\)/($1 * interval '1 second')/igs; + $str =~ s/\bSTR_TO_DATE\(\s*(.*?)\s*,\s*('[^'\(\),]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_strtodate_to_pgsql($class, $1, $2)/eigs; + $str =~ s/\b(SUBDATE|DATE_SUB)\(\s*([^,]+)\s*,\s*INTERVAL ([^\(\)]+)\s*\)/($2)::timestamp - interval '$3'/igs; + $str =~ s/\bSUBDATE\(\s*([^,]+)\s*,\s*(\d+)\s*\)/($1)::timestamp - ($2 * interval '1 day')/igs; + $str =~ s/\bSUBTIME\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp - ($2)::interval/igs; + $str =~ s/\bTIME(\([^\(\)]+\))/($1)::time/igs; + $str =~ s/\bTIMEDIFF\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp - ($2)::timestamp/igs; + $str =~ s/\bTIMESTAMP\(\s*([^\(\)]+)\s*\)/($1)::timestamp/igs; + $str =~ s/\bTIMESTAMP\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($1)::timestamp + ($2)::time/igs; + $str =~ s/\bTIMESTAMPADD\(\s*([^,]+)\s*,\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/($3)::timestamp + ($1 * interval '1 $2')/igs; + $str =~ s/\bTIMESTAMPDIFF\(\s*YEAR\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/extract(year from ($2)::timestamp) - extract(year from ($1)::timestamp)/igs; + $str =~ s/\bTIMESTAMPDIFF\(\s*MONTH\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/(extract(year from ($2)::timestamp) - extract(year from ($1)::timestamp))*12 + (extract(month from ($2)::timestamp) - extract(month from ($1)::timestamp))/igs; + $str =~ s/\bTIMESTAMPDIFF\(\s*WEEK\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/floor(extract(day from ( ($2)::timestamp - ($1)::timestamp))\/7)/igs; + $str =~ s/\bTIMESTAMPDIFF\(\s*DAY\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/extract(day from ( ($2)::timestamp - ($1)::timestamp))/igs; + $str =~ s/\bTIMESTAMPDIFF\(\s*HOUR\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/floor(extract(epoch from ( ($2)::timestamp - ($1)::timestamp))\/3600)/igs; + $str =~ s/\bTIMESTAMPDIFF\(\s*MINUTE\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/floor(extract(epoch from ( ($2)::timestamp - ($1)::timestamp))\/60)/igs; + $str =~ s/\bTIMESTAMPDIFF\(\s*SECOND\s*,\s*([^,]+)\s*,\s*([^\(\),]+)\s*\)/extract(epoch from ($2)::timestamp) - extract(epoch from ($1)::timestamp))/igs; + $str =~ s/\bTIME_FORMAT\(\s*(.*?)\s*,\s*('[^'\(\),]+'|\?TEXTVALUE\d+\?)\s*\)/_mysql_timeformat_to_pgsql($class, $1, $2)/eigs; + $str =~ s/\bTIME_TO_SEC\(\s*([^\(\)]+)\s*\)/(extract(hours from ($1)::time)*3600 + extract(minutes from ($1)::time)*60 + extract(seconds from ($1)::time))::bigint/igs; + $str =~ s/\bTO_DAYS\(\s*([^\(\)]+)\s*\)/(($1)::date - '0001-01-01bc')::integer/igs; + $str =~ s/\bWEEK(\([^\(\)]+\))/extract(week from date($1)) - 1/igs; + $str =~ s/\bWEEKOFYEAR(\([^\(\)]+\))/extract(week from date($2))/igs; + $str =~ s/\bWEEKDAY\(\s*([^\(\)]+)\s*\)/to_char(($1)::timestamp, 'ID')::integer - 1/igs; # MySQL: Monday = 0, PG => 1 + $str =~ s/\bYEAR\(\s*([^\(\)]+)\s*\)/extract(year from date($1))/igs; + + # String functions + $str =~ s/\bBIN\(\s*([^\(\)]+)\s*\)/ltrim(textin(bit_out($1::bit(64))), '0')/igs; + $str =~ s/\bBINARY\(\s*([^\(\)]+)\s*\)/($1)::bytea/igs; + $str =~ s/\bBIT_COUNT\(\s*([^\(\)]+)\s*\)/length(replace(ltrim(textin(bit_out($1::bit(64))),'0'),'0',''))/igs; + $str =~ s/\bCHAR\(\s*([^\(\),]+)\s*\)/array_to_string(ARRAY(SELECT chr(unnest($1))),'')/igs; + $str =~ s/\bELT\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(ARRAY[$2])[$1]/igs; + $str =~ s/\bFIELD\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(SELECT i FROM generate_subscripts(array[$2], 1) g(i) WHERE $1 = (array[$2])[i] UNION ALL SELECT 0 LIMIT 1)/igs; + $str =~ s/\bFIND_IN_SET\(\s*([^,]+)\s*,\s*([^\(\)]+)\s*\)/(SELECT i FROM generate_subscripts(string_to_array($2,','), 1) g(i) WHERE $1 = (string_to_array($2,','))[i] UNION ALL SELECT 0 LIMIT 1)/igs; + $str =~ s/\bFROM_BASE64\(\s*([^\(\),]+)\s*\)/decode(($1)::bytea, 'base64')/igs; + $str =~ s/\bHEX\(\s*([^\(\),]+)\s*\)/upper(encode($1::bytea, 'hex'))/igs; + $str =~ s/\bINSTR\s*\(\s*([^,]+),\s*('[^']+')\s*\)/position($2 in $1)/igs; + if (!$class->{pg_supports_substr}) { + $str =~ s/\bLOCATE\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/position($1 in substring ($2 from $3)) + $3 - 1/igs; + $str =~ s/\bMID\(/substring\(/igs; + } else { + $str =~ s/\bLOCATE\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/position($1 in substr($2, $3)) + $3 - 1/igs; + $str =~ s/\bMID\(/substr\(/igs; + } + $str =~ s/\bLOCATE\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/position($1 in $2)/igs; + $str =~ s/\bLCASE\(/lower\(/igs; + $str =~ s/\bORD\(/ascii\(/igs; + $str =~ s/\bQUOTE\(/quote_literal\(/igs; + $str =~ s/\bSPACE\(\s*([^\(\),]+)\s*\)/repeat(' ', $1)/igs; + $str =~ s/\bSTRCMP\(\s*([^\(\),]+)\s*,\s*([^\(\),]+)\s*\)/CASE WHEN $1 < $2 THEN -1 WHEN $1 > $2 THEN 1 ELSE 0 END/igs; + $str =~ s/\bTO_BASE64\(\s*([^\(\),]+)\s*\)/encode($1, 'base64')/igs; + $str =~ s/\bUCASE\(/upper\(/igs; + $str =~ s/\bUNHEX\(\s*([^\(\),]+)\s*\)/decode($1, 'hex')::text/igs; + $str =~ s/\bIS_IPV6\(\s*([^\(\)]+)\s*\)/CASE WHEN family($1) = 6 THEN 1 ELSE 0 END/igs; + $str =~ s/\bIS_IPV4\(\s*([^\(\)]+)\s*\)/CASE WHEN family($1) = 4 THEN 1 ELSE 0 END/igs; + $str =~ s/\bISNULL\(\s*([^\(\)]+)\s*\)/$1 IS NULL/igs; + $str =~ s/\bRLIKE/REGEXP/igs; + $str =~ s/\bSTD\(/STDDEV_POP\(/igs; + $str =~ s/\bSTDDEV\(/STDDEV_POP\(/igs; + $str =~ s/\bUUID\(/$class->{uuid_function}\(/igs; + $str =~ s/\bNOT REGEXP BINARY/\!\~/igs; + $str =~ s/\bREGEXP BINARY/\~/igs; + $str =~ s/\bNOT REGEXP/\!\~\*/igs; + $str =~ s/\bREGEXP/\~\*/igs; + + $str =~ s/\bGET_LOCK/pg_advisory_lock/igs; + $str =~ s/\bIS_USED_LOCK/pg_try_advisory_lock/igs; + $str =~ s/\bRELEASE_LOCK/pg_advisory_unlock/igs; + + # GROUP_CONCAT doesn't exist, it must be replaced by calls to array_to_string() and array_agg() functions + $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s+(ASC|DESC)\s+SEPARATOR\s+(\?TEXTVALUE\d+\?|'[^']+')\s*\)/array_to_string(array_agg($1 ORDER BY $2 $3), $4)/igs; + $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s+SEPARATOR\s+(\?TEXTVALUE\d+\?|'[^']+')\s*\)/array_to_string(array_agg($1 ORDER BY $2 ASC), $3)/igs; + $str =~ s/GROUP_CONCAT\((.*?)\s+SEPARATOR\s+(\?TEXTVALUE\d+\?|'[^']+')\s*\)/array_to_string(array_agg($1), $2)/igs; + $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s+(ASC|DESC)\s*\)/array_to_string(array_agg($1 ORDER BY $2 $3), ',')/igs; + $str =~ s/GROUP_CONCAT\((.*?)\s+ORDER\s+BY\s+([^\s]+)\s*\)/array_to_string(array_agg($1 ORDER BY $2), ',')/igs; + $str =~ s/GROUP_CONCAT\(([^\)]+)\)/array_to_string(array_agg($1), ',')/igs; + + # Replace IFNULL() MySQL function in a query + while ($str =~ s/\bIFNULL\(\s*([^,]+)\s*,\s*([^\)]+\s*)\)/COALESCE($1, $2)/is) {}; + + # Rewrite while loop + $str =~ s/\bWHILE\b(.*?)\bEND\s+WHILE\s*;/WHILE $1END LOOP;/igs; + $str =~ s/\bWHILE\b(.*?)\bDO\b/WHILE $1LOOP/igs; + + # Rewrite REPEAT loop + my %repl_repeat = (); + $i = 0; + while ($str =~ s/\bREPEAT\s+(.*?)\bEND REPEAT\s*;/%REPREPEATLBL$i%/igs) { + my $code = $1; + $code =~ s/\bUNTIL(.*)//; + $repl_repeat{$i} = "LOOP ${code}EXIT WHEN $1;\nEND LOOP;"; + } + foreach $i (keys %repl_repeat) { + $str =~ s/\%REPREPEATLBL$i\%/$repl_repeat{$i}/gs; + } + %repl_repeat = (); + + # Fix some charset encoding call in cast function + #$str =~ s/(CAST\s*\((?:.*?)\s+AS\s+(?:[^\s]+)\s+)CHARSET\s+([^\s\)]+)\)/$1) COLLATE "\U$2\E"/igs; + $str =~ s/(CAST\s*\((?:.*?)\s+AS\s+(?:[^\s]+)\s+)(CHARSET|CHARACTER\s+SET)\s+([^\s\)]+)\)/$1)/igs; + $str =~ s/CONVERT\s*(\((?:[^,]+)\s+,\s+(?:[^\s]+)\s+)(CHARSET|CHARACTER\s+SET)\s+([^\s\)]+)\)/CAST$1)/igs; + $str =~ s/CONVERT\s*\((.*?)\s+USING\s+([^\s\)]+)\)/CAST($1 AS text)/igs; + # Set default UTF8 collation to postgreSQL equivalent C.UTF-8 + #$str =~ s/COLLATE "UTF8"/COLLATE "C.UTF-8"/gs; + $str =~ s/\bCHARSET(\s+)/COLLATE$1/igs; + + # Remove call to start transaction + $str =~ s/\sSTART\s+TRANSACTION\s*;/-- START TRANSACTION;/igs; + + # Comment call to COMMIT or ROLLBACK in the code if allowed + if ($class->{comment_commit_rollback}) { + $str =~ s/\b(COMMIT|ROLLBACK)\s*;/-- $1;/igs; + $str =~ s/(ROLLBACK\s+TO\s+[^;]+);/-- $1;/igs; + } + + # Translate call to CREATE TABLE ... SELECT + $str =~ s/CREATE\s+PRIVATE\s+TEMPORARY/CREATE TEMPORARY/; + $str =~ s/(CREATE(?:\s+TEMPORARY)?\s+TABLE\s+[^\s]+)(\s+SELECT)/$1 AS $2/igs; + $str =~ s/ON\s+COMMIT\s+PRESERVE\s+DEFINITION/ON COMMIT PRESERVE ROWS/igs; + $str =~ s/ON\s+COMMIT\s+DROP\s+DEFINITION/ON COMMIT DROP/igs; + + # Remove @ from variables and rewrite SET assignement in QUERY mode + if ($class->{type} eq 'QUERY') { + $str =~ s/\@([^\s]+)\b/$1/gs; + $str =~ s/:=/=/gs; + } + + # Replace spatial related lines + $str = replace_mysql_spatial($str); + + # Rewrite direct call to function without out parameters using PERFORM + $str = perform_replacement($class, $str); + + # Remove CALL from all statements if not supported + if (!$class->{pg_supports_procedure}) { + $str =~ s/\bCALL\s+//igs; + } + + return $str; +} + +sub _replace_dateadd +{ + my $str = shift; + my $dd = shift; + + my $op = '+'; + if ($str =~ s/^\-[\s]*//) { + $op = '-'; + } + if ($str =~ s/^(\d+)\s+([^\(\),\s]+)$/ $op $1*interval '1 $2'/s) { + return $str; + } elsif ($str =~ s/^([^\s]+)\s+([^\(\),\s]+)$/ $op $1*interval '1 $2'/s) { + return $str; + } elsif ($str =~ s/^([^\(\),]+)$/ $op interval '$1'/s) { + return $str; + } + + return $str; +} + + +sub replace_mysql_spatial +{ + my $str = shift; + + $str =~ s/AsWKB\(/AsBinary\(/igs; + $str =~ s/AsWKT\(/AsText\(/igs; + $str =~ s/GeometryCollectionFromText\(/GeomCollFromText\(/igs; + $str =~ s/GeometryCollectionFromWKB\(/GeomCollFromWKB\(/igs; + $str =~ s/GeometryFromText\(/GeomFromText\(/igs; + $str =~ s/GLength\(/ST_Length\(/igs; + $str =~ s/LineStringFromWKB\(/LineFromWKB\(/igs; + $str =~ s/MultiLineStringFromText\(/MLineFromText\(/igs; + $str =~ s/MultiPointFromText\(/MPointFromText\(/igs; + $str =~ s/MultiPolygonFromText\(/MPolyFromText\(/igs; + $str =~ s/PolyFromText\(/PolygonFromText\(/igs; + $str =~ s/MBRContains\(/ST_Contains\(/igs; + $str =~ s/MBRDisjoint\(/ST_Disjoint\(/igs; + $str =~ s/MBREqual\(/ST_Equals\(/igs; + $str =~ s/MBRIntersects\(/ST_Intersects\(/igs; + $str =~ s/MBROverlaps\(/ST_Overlaps\(/igs; + $str =~ s/MBRTouches\(/ST_Touches\(/igs; + $str =~ s/MBRWithin\(/ST_Within\(/igs; + $str =~ s/MLineFromWKB\(/MultiLineStringFromWKB\(/igs; + $str =~ s/MPointFromWKB\(/MultiPointFromWKB\(/igs; + $str =~ s/MPolyFromWKB\(/MultiPolygonFromWKB\(/igs; + $str =~ s/PolyFromWKB\(/PolygonFromWKB\(/igs; + + # Replace FromWKB functions + foreach my $fct ('MultiLineStringFromWKB', 'MultiPointFromWKB', 'MultiPolygonFromWKB', 'PolygonFromWKB') { + $str =~ s/\b$fct\(/ST_GeomFromWKB\(/igs; + } + + # Add ST_ prefix to function alias + foreach my $fct (@MYSQL_SPATIAL_FCT) { + $str =~ s/\b$fct\(/ST_$fct\(/igs; + } + + return $str; +} + +sub _mysql_getformat_to_pgsql +{ + my ($type, $format) = @_; + + if (uc($type) eq 'DATE') { + if (uc($format) eq "'USA'") { + $format = "'%m.%d.%Y'"; + } elsif (uc($format) eq "'EUR'") { + $format = "'%d.%m.%Y'"; + } elsif (uc($format) eq "'INTERNAL'") { + $format = "'%Y%m%d'"; + } else { + # ISO and JIS + $format = "'%Y-%m-%d'"; + } + } elsif (uc($type) eq 'TIME') { + if (uc($format) eq "'USA'") { + $format = "'%h:%i:%s %p'"; + } elsif (uc($format) eq "'EUR'") { + $format = "'%H.%i.%s'"; + } elsif (uc($format) eq "'INTERNAL'") { + $format = "'%H%i%s'"; + } else { + # ISO and JIS + $format = "'%H:%i:%s'"; + } + } else { + if ( (uc($format) eq "'USA'") || (uc($format) eq "'EUR'") ) { + $format = "'%Y-%m-%d %H.%i.%s'"; + } elsif (uc($format) eq "'INTERNAL'") { + $format = "'%Y%m%d%H%i%s'"; + } else { + # ISO and JIS + $format = "'%Y-%m-%d %H:%i:%s'"; + } + } + + return $format; +} + +sub _mysql_strtodate_to_pgsql +{ + my ($class, $datetime, $format) = @_; + + my $str = _mysql_dateformat_to_pgsql($class, $datetime, $format, 1); + + return $str; +} + +sub _mysql_timeformat_to_pgsql +{ + my ($class, $datetime, $format) = @_; + + my $str = _mysql_dateformat_to_pgsql($class, $datetime, $format, 0, 1); + + return $str; +} + + +sub _mysql_dateformat_to_pgsql +{ + my ($class, $datetime, $format, $todate, $totime) = @_; + +# Not supported: +# %X Year for the week where Sunday is the first day of the week, numeric, four digits; used with %V + + if ($format =~ s/\?TEXTVALUE(\d+)\?/$class->{text_values}{$1}/) { + delete $class->{text_values}{$1}; + } + + $format =~ s/\%a/Dy/g; + $format =~ s/\%b/Mon/g; + $format =~ s/\%c/FMMM/g; + $format =~ s/\%D/FMDDth/g; + $format =~ s/\%e/FMDD/g; + $format =~ s/\%f/US/g; + $format =~ s/\%H/HH24/g; + $format =~ s/\%h/HH12/g; + $format =~ s/\%I/HH/g; + $format =~ s/\%i/MI/g; + $format =~ s/\%j/DDD/g; + $format =~ s/\%k/FMHH24/g; + $format =~ s/\%l/FMHH12/g; + $format =~ s/\%m/MM/g; + $format =~ s/\%p/AM/g; + $format =~ s/\%r/HH12:MI:SS AM/g; + $format =~ s/\%s/SS/g; + $format =~ s/\%S/SS/g; + $format =~ s/\%T/HH24:MI:SS/g; + $format =~ s/\%U/WW/g; + $format =~ s/\%u/IW/g; + $format =~ s/\%V/WW/g; + $format =~ s/\%v/IW/g; + $format =~ s/\%x/YYYY/g; + $format =~ s/\%X/YYYY/g; + $format =~ s/\%Y/YYYY/g; + $format =~ s/\%y/YY/g; + $format =~ s/\%W/Day/g; + $format =~ s/\%M/Month/g; + $format =~ s/\%(\d+)/$1/g; + + # Replace constant strings + if ($format =~ s/('[^']+')/\?TEXTVALUE$class->{text_values_pos}\?/s) { + $class->{text_values}{$class->{text_values_pos}} = $1; + $class->{text_values_pos}++; + } + + if ($todate) { + return "to_date($datetime, $format)"; + } elsif ($totime) { + return "to_char(($datetime)::time, $format)"; + } + + return "to_char(($datetime)::timestamp, $format)"; +} + +sub mysql_estimate_cost +{ + my $str = shift; + my $type = shift; + + my %cost_details = (); + + # Default cost is testing that mean it at least must be tested + my $cost = $FCT_TEST_SCORE; + # When evaluating queries tests must not be included here + if ($type eq 'QUERY') { + $cost = 0; + } + $cost_details{'TEST'} = $cost; + + # Set cost following code length + my $cost_size = int(length($str)/$SIZE_SCORE) || 1; + # When evaluating queries size must not be included here + if ($type eq 'QUERY') { + $cost_size = 0; + } + + $cost += $cost_size; + $cost_details{'SIZE'} = $cost_size; + + # Try to figure out the manual work + my $n = () = $str =~ m/(ARRAY_AGG|GROUP_CONCAT)\(\s*DISTINCT/igs; + $cost_details{'ARRAY_AGG_DISTINCT'} += $n; + $n = () = $str =~ m/\bSOUNDS\s+LIKE\b/igs; + $cost_details{'SOUNDS LIKE'} += $n; + $n = () = $str =~ m/CHARACTER\s+SET/igs; + $cost_details{'CHARACTER SET'} += $n; + $n = () = $str =~ m/\bCOUNT\(\s*DISTINCT\b/igs; + $cost_details{'COUNT(DISTINCT)'} += $n; + $n = () = $str =~ m/\bMATCH.*AGAINST\b/igs; + $cost_details{'MATCH'} += $n; + $n = () = $str =~ m/\bJSON_[A-Z\_]+\(/igs; + $cost_details{'JSON FUNCTION'} += $n; + $n = () = $str =~ m/_(un)?lock\(/igs; + $cost_details{'LOCK'} += $n; + $n = () = $str =~ m/\b\@+[A-Z0-9\_]+/igs; + $cost_details{'@VAR'} += $n; + + foreach my $t (keys %UNCOVERED_MYSQL_SCORE) { + $cost += $UNCOVERED_MYSQL_SCORE{$t}*$cost_details{$t}; + } + foreach my $f (@MYSQL_FUNCTIONS) { + if ($str =~ /\b$f\b/igs) { + $cost += 2; + $cost_details{$f} += 2; + } + } + + return $cost, %cost_details; +} + +sub replace_outer_join +{ + my ($class, $str, $type) = @_; + + if (!grep(/^$type$/, 'left', 'right')) { + die "FATAL: outer join type must be 'left' or 'right' in call to replace_outer_join().\n"; + } + + # When we have a right outer join, just rewrite it as a left join to simplify the translation work + if ($type eq 'right') { + $str =~ s/(\s+)([^\s]+)\s*(\%OUTERJOIN\d+\%)\s*(!=|<>|>=|<=|=|>|<|NOT LIKE|LIKE)\s*([^\s]+)/$1$5 $4 $2$3/isg; + return $str; + } + + my $regexp1 = qr/((?:!=|<>|>=|<=|=|>|<|NOT LIKE|LIKE)\s*[^\s]+\s*\%OUTERJOIN\d+\%)/is; + my $regexp2 = qr/\%OUTERJOIN\d+\%\s*(?:!=|<>|>=|<=|=|>|<|NOT LIKE|LIKE)/is; + + # process simple form of outer join + my $nbouter = $str =~ $regexp1; + + # Check that we don't have right outer join too + if ($nbouter >= 1 && $str !~ $regexp2) + { + # Extract tables in the FROM clause + $str =~ s/(.*)\bFROM\s+(.*?)\s+WHERE\s+(.*?)$/$1FROM FROM_CLAUSE WHERE $3/is; + my $from_clause = $2; + $from_clause =~ s/"//gs; + my @tables = split(/\s*,\s*/, $from_clause); + + # Set a hash for alias to table mapping + my %from_clause_list = (); + my %from_order = (); + my $fidx = 0; + foreach my $table (@tables) + { + $table =~ s/^\s+//s; + $table =~ s/\s+$//s; + my $cmt = ''; + while ($table =~ s/(\s*\%ORA2PG_COMMENT\d+\%\s*)//is) { + $cmt .= $1; + } + my ($t, $alias, @others) = split(/\s+/, lc($table)); + $alias = $others[0] if (uc($alias) eq 'AS'); + $alias = "$t" if (!$alias); + $from_clause_list{$alias} = "$cmt$t"; + $from_order{$alias} = $fidx++; + } + + # Extract all Oracle's outer join syntax from the where clause + my @outer_clauses = (); + my %final_outer_clauses = (); + my %final_from_clause = (); + my @tmp_from_list = (); + my $start_query = ''; + my $end_query = ''; + if ($str =~ s/^(.*FROM FROM_CLAUSE WHERE)//is) { + $start_query = $1; + } + if ($str =~ s/\s+((?:START WITH|CONNECT BY|ORDER SIBLINGS BY|GROUP BY|ORDER BY).*)$//is) { + $end_query = $1; + } + + # Extract predicat from the WHERE clause + my @predicat = split(/\s*(\bAND\b|\bOR\b|\%ORA2PG_COMMENT\d+\%)\s*/i, $str); + my $id = 0; + my %other_join_clause = (); + # Process only predicat with a obsolete join syntax (+) for now + for (my $i = 0; $i <= $#predicat; $i++) + { + next if ($predicat[$i] !~ /\%OUTERJOIN\d+\%/i); + my $where_clause = $predicat[$i]; + $where_clause =~ s/"//gs; + $where_clause =~ s/^\s+//s; + $where_clause =~ s/[\s;]+$//s; + $where_clause =~ s/\s*(\%OUTERJOIN\d+\%)//gs; + + $predicat[$i] = "WHERE_CLAUSE$id "; + + # Split the predicat to retrieve left part, operator and right part + my ($l, $o, $r) = split(/\s*(!=|>=|<=|=|<>|<|>|NOT LIKE|LIKE)\s*/i, $where_clause); + + # NEW / OLD pseudo table in triggers can not be part of a join + # clause. Move them int to the WHERE clause. + if ($l =~ /^(NEW|OLD)\./is) + { + $predicat[$i] =~ s/WHERE_CLAUSE$id / $l $o $r /s; + next; + } + $id++; + + # Extract the tablename part of the left clause + my $lbl1 = ''; + my $table_decl1 = $l; + if ($l =~ /^([^\.\s]+\.[^\.\s]+)\..*/ || $l =~ /^([^\.\s]+)\..*/) + { + $lbl1 = lc($1); + # If the table/alias is not part of the from clause + if (!exists $from_clause_list{$lbl1}) { + $from_clause_list{$lbl1} = $lbl1; + $from_order{$lbl1} = $fidx++; + } + $table_decl1 = $from_clause_list{$lbl1}; + $table_decl1 .= " $lbl1" if ($lbl1 ne $from_clause_list{$lbl1}); + } + elsif ($l =~ /\%SUBQUERY(\d+)\%/) + { + # Search for table.column in the subquery or function code + my $tmp_str = $l; + while ($tmp_str =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) + { + if ($tmp_str =~ /\b([^\.\s]+\.[^\.\s]+)\.[^\.\s]+/ + || $tmp_str =~ /\b([^\.\s]+)\.[^\.\s]+/) + { + $lbl1 = lc($1); + # If the table/alias is not part of the from clause + if (!exists $from_clause_list{$lbl1}) + { + $from_clause_list{$lbl1} = $lbl1; + $from_order{$lbl1} = $fidx++; + } + $table_decl1 = $from_clause_list{$lbl1}; + $table_decl1 .= " $lbl1" if ($lbl1 ne $from_clause_list{$lbl1}); + last; + } + } + } + + # Extract the tablename part of the right clause + my $lbl2 = ''; + my $table_decl2 = $r; + if ($r =~ /^([^\.\s]+\.[^\.\s]+)\..*/ || $r =~ /^([^\.\s]+)\..*/) + { + $lbl2 = lc($1); + if (!$lbl1) { + push(@{$other_join_clause{$lbl2}}, "$l $o $r"); + next; + } + # If the table/alias is not part of the from clause + if (!exists $from_clause_list{$lbl2}) { + $from_clause_list{$lbl2} = $lbl2; + $from_order{$lbl2} = $fidx++; + } + $table_decl2 = $from_clause_list{$lbl2}; + $table_decl2 .= " $lbl2" if ($lbl2 ne $from_clause_list{$lbl2}); + } + elsif ($lbl1) + { + # Search for table.column in the subquery or function code + my $tmp_str = $r; + while ($tmp_str =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) + { + if ($tmp_str =~ /\b([^\.\s]+\.[^\.\s]+)\.[^\.\s]+/ + || $tmp_str =~ /\b([^\.\s]+)\.[^\.\s]+/) + { + $lbl2 = lc($1); + # If the table/alias is not part of the from clause + if (!exists $from_clause_list{$lbl2}) + { + $from_clause_list{$lbl2} = $lbl2; + $from_order{$lbl2} = $fidx++; + } + $table_decl2 = $from_clause_list{$lbl2}; + $table_decl2 .= " $lbl2" if ($lbl2 ne $from_clause_list{$lbl2}); + } + } + if (!$lbl2 ) + { + push(@{$other_join_clause{$lbl1}}, "$l $o $r"); + next; + } + } + + # When this is the first join parse add the left tablename + # first then the outer join with the right table + if (scalar keys %final_from_clause == 0) + { + $from_clause = $table_decl1; + $table_decl1 =~ s/\s*\%ORA2PG_COMMENT\d+\%\s*//igs; + push(@outer_clauses, (split(/\s/, $table_decl1))[1] || $table_decl1); + $final_from_clause{"$lbl1;$lbl2"}{position} = $i; + push(@{$final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{predicat}}, "$l $o $r"); + } + else + { + $final_from_clause{"$lbl1;$lbl2"}{position} = $i; + push(@{$final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{predicat}}, "$l $o $r"); + if (!exists $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{$type}) { + $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{$type} = $table_decl1; + } + } + if ($type eq 'left') { + $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl2}{position} = $i; + } else { + $final_from_clause{"$lbl1;$lbl2"}{clause}{$table_decl1}{position} = $i; + } + } + $str = $start_query . join(' ', @predicat) . ' ' . $end_query; + + # Remove part from the WHERE clause that will be moved into the FROM clause + $str =~ s/\s*(AND\s+)?WHERE_CLAUSE\d+ / /igs; + $str =~ s/WHERE\s+(AND|OR)\s+/WHERE /is; + $str =~ s/WHERE[\s;]+$//i; + $str =~ s/(\s+)WHERE\s+(ORDER|GROUP)\s+BY/$1$2 BY/is; + $str =~ s/\s+WHERE(\s+)/\nWHERE$1/igs; + + my %associated_clause = (); + foreach my $t (sort { $final_from_clause{$a}{position} <=> $final_from_clause{$b}{position} } keys %final_from_clause) + { + foreach my $j (sort { $final_from_clause{$t}{clause}{$a}{position} <=> $final_from_clause{$t}{clause}{$b}{position} } keys %{$final_from_clause{$t}{clause}}) + { + next if ($#{$final_from_clause{$t}{clause}{$j}{predicat}} < 0); + + if (exists $final_from_clause{$t}{clause}{$j}{$type} && $j !~ /\%SUBQUERY\d+\%/i && $from_clause !~ /\b\Q$final_from_clause{$t}{clause}{$j}{$type}\E\b/) + { + $from_clause .= ",$final_from_clause{$t}{clause}{$j}{$type}"; + push(@outer_clauses, (split(/\s/, $final_from_clause{$t}{clause}{$j}{$type}))[1] || $final_from_clause{$t}{clause}{$j}{$type}); + } + my ($l,$r) = split(/;/, $t); + my $tbl = $j; + $tbl =~ s/\s*\%ORA2PG_COMMENT\d+\%\s*//isg; + $from_clause .= "\n\U$type\E OUTER JOIN $tbl ON (" . join(' AND ', @{$final_from_clause{$t}{clause}{$j}{predicat}}) . ")"; + push(@{$final_outer_clauses{$l}{join}}, "\U$type\E OUTER JOIN $tbl ON (" . join(' AND ', @{$final_from_clause{$t}{clause}{$j}{predicat}}, @{$other_join_clause{$r}}) . ")"); + push(@{$final_outer_clauses{$l}{position}}, $final_from_clause{$t}{clause}{$j}{position}); + push(@{$associated_clause{$l}}, $r); + } + } + + $from_clause = ''; + my @clause_done = (); + foreach my $c (sort { $from_order{$a} <=> $from_order{$b} } keys %from_order) + { + next if (!grep(/^\Q$c\E$/i, @outer_clauses)); + my @output = (); + for (my $j = 0; $j <= $#{$final_outer_clauses{$c}{join}}; $j++) { + push(@output, $final_outer_clauses{$c}{join}[$j]); + } + + find_associated_clauses($c, \@output, \%associated_clause, \%final_outer_clauses); + + if (!grep(/\QJOIN $from_clause_list{$c} $c \E/is, @clause_done)) + { + $from_clause .= "\n, $from_clause_list{$c}"; + $from_clause .= " $c" if ($c ne $from_clause_list{$c}); + } + foreach (@output) { + $from_clause .= "\n" . $_; + } + push(@clause_done, @output); + delete $from_order{$c}; + delete $final_outer_clauses{$c}; + delete $associated_clause{$c}; + } + $from_clause =~ s/^\s*,\s*//s; + + # Append tables to from clause that was not involved into an outer join + foreach my $a (sort keys %from_clause_list) + { + my $table_decl = "$from_clause_list{$a}"; + $table_decl .= " $a" if ($a ne $from_clause_list{$a}); + # Remove comment before searching it inside the from clause + my $tmp_tbl = $table_decl; + my $comment = ''; + while ($tmp_tbl =~ s/(\s*\%ORA2PG_COMMENT\d+\%\s*)//is) { + $comment .= $1; + } + + if ($from_clause !~ /(^|\s|,)\Q$tmp_tbl\E\b/is) { + $from_clause = "$table_decl, " . $from_clause; + } elsif ($comment) { + $from_clause = "$comment " . $from_clause; + } + } + $from_clause =~ s/\b(new|old)\b/\U$1\E/gs; + $from_clause =~ s/,\s*$/ /s; + $str =~ s/FROM FROM_CLAUSE/FROM $from_clause/s; + } + + return $str; +} + +sub find_associated_clauses +{ + my ($c, $output, $associated_clause, $final_outer_clauses) = @_; + + foreach my $f (@{$associated_clause->{$c}}) { + for (my $j = 0; $j <= $#{$final_outer_clauses->{$f}{join}}; $j++) { + push(@$output, $final_outer_clauses->{$f}{join}[$j]); + } + delete $final_outer_clauses->{$f}; + find_associated_clauses($f, $output, $associated_clause, $final_outer_clauses); + } + delete $associated_clause->{$c}; +} + + +sub replace_connect_by +{ + my ($class, $str) = @_; + + return $str if ($str !~ /\bCONNECT\s+BY\b/is); + + my $final_query = "WITH RECURSIVE cte AS (\n"; + + # Remove NOCYCLE, not supported at now + $str =~ s/\s+NOCYCLE//is; + + # Remove SIBLINGS keywords and enable siblings rewrite + my $siblings = 0; + if ($str =~ s/\s+SIBLINGS//is) { + $siblings = 1; + } + + # Extract UNION part of the query to past it at end + my $union = ''; + if ($str =~ s/(CONNECT BY.*)(\s+UNION\s+.*)/$1/is) { + $union = $2; + } + + # Extract order by to past it to the query at end + my $order_by = ''; + if ($str =~ s/\s+ORDER BY(.*)//is) { + $order_by = $1; + } + + # Extract group by to past it to the query at end + my $group_by = ''; + if ($str =~ s/(\s+GROUP BY.*)//is) { + $group_by = $1; + } + + # Extract the starting node or level of the tree + my $where_clause = ''; + my $start_with = ''; + if ($str =~ s/WHERE\s+(.*?)\s+START\s+WITH\s*(.*?)\s+CONNECT BY\s*//is) { + $where_clause = " WHERE $1"; + $start_with = $2; + } elsif ($str =~ s/WHERE\s+(.*?)\s+CONNECT BY\s+(.*?)\s+START\s+WITH\s*(.*)/$2/is) { + $where_clause = " WHERE $1"; + $start_with = $3; + } elsif ($str =~ s/START\s+WITH\s*(.*?)\s+CONNECT BY\s*//is) { + $start_with = $1; + } elsif ($str =~ s/\s+CONNECT BY\s+(.*?)\s+START\s+WITH\s*(.*)/ $1 /is) { + $start_with = $2; + } else { + $str =~ s/CONNECT BY\s*//is; + } + + # remove alias from where clause + $where_clause =~ s/\b[^\.]\.([^\s]+)\b/$1/gs; + + # Extract the CONNECT BY clause in the hierarchical query + my $prior_str = ''; + my @prior_clause = ''; + if ($str =~ s/([^\s]+\s*=\s*PRIOR\s+.*)//is) { + $prior_str = $1; + } elsif ($str =~ s/(\s*PRIOR\s+.*)//is) { + $prior_str = $1; + } else { + # look inside subqueries if we have a prior clause + my @ids = $str =~ /\%SUBQUERY(\d+)\%/g; + my $sub_prior_str = ''; + foreach my $i (@ids) { + if ($class->{sub_parts}{$i} =~ s/([^\s]+\s*=\s*PRIOR\s+.*)//is) { + $sub_prior_str = $1; + $str =~ s/\%SUBQUERY$i\%//; + } elsif ($class->{sub_parts}{$i} =~ s/(\s*PRIOR\s+.*)//is) { + $sub_prior_str = $1; + $str =~ s/\%SUBQUERY$i\%//; + } + $sub_prior_str =~ s/^\(//; + $sub_prior_str =~ s/\)$//; + ($prior_str ne '' || $sub_prior_str eq '') ? $prior_str .= ' ' . $sub_prior_str : $prior_str = $sub_prior_str; + } + } + if ($prior_str) { + # Try to extract the prior clauses + my @tmp_prior = split(/\s*AND\s*/, $prior_str); + $tmp_prior[-1] =~ s/\s*;\s*//s; + my @tmp_prior2 = (); + foreach my $p (@tmp_prior) { + if ($p =~ /\bPRIOR\b/is) { + push(@prior_clause, split(/\s*=\s*/i, $p)); + } else { + $where_clause .= " AND $p"; + } + } + if ($siblings) { + if ($prior_clause[-1] !~ /PRIOR/i) { + $siblings = $prior_clause[-1]; + } else { + $siblings = $prior_clause[-2]; + } + $siblings =~ s/\s+//g; + } + shift(@prior_clause) if ($prior_clause[0] eq ''); + my @rebuild_prior = (); + # Place PRIOR in the left part if necessary + for (my $i = 0; $i < $#prior_clause; $i+=2) { + if ($prior_clause[$i+1] =~ /PRIOR\s+/i) { + my $tmp = $prior_clause[$i]; + $prior_clause[$i] = $prior_clause[$i+1]; + $prior_clause[$i+1] = $tmp; + } + push(@rebuild_prior, "$prior_clause[$i] = $prior_clause[$i+1]"); + } + @prior_clause = @rebuild_prior; + # Remove table aliases from prior clause + map { s/\s*PRIOR\s*//s; s/[^\s\.=<>!]+\.//s; } @prior_clause; + } + my $bkup_query = $str; + # Construct the initialization query + $str =~ s/(SELECT\s+)(.*?)(\s+FROM)/$1COLUMN_ALIAS$3/is; + my @columns = split(/\s*,\s*/, $2); + # When the pseudo column LEVEL is used in the where clause + # and not used in columns list, add the pseudo column + if ($where_clause =~ /\bLEVEL\b/is && !grep(/\bLEVEL\b/i, @columns)) { + push(@columns, 'level'); + } + my @tabalias = (); + my %connect_by_path = (); + for (my $i = 0; $i <= $#columns; $i++) { + my $found = 0; + while ($columns[$i] =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) { + # Get out of here next run when a call to SYS_CONNECT_BY_PATH is found + # This will prevent opening too much subquery in the function parameters + last if ($found); + $found = 1 if ($columns[$i]=~ /SYS_CONNECT_BY_PATH/is); + }; + # Replace LEVEL call by a counter, there is no direct equivalent in PostgreSQL + if (lc($columns[$i]) eq 'level') { + $columns[$i] = "1 as level"; + } elsif ($columns[$i] =~ /\bLEVEL\b/is) { + $columns[$i] =~ s/\bLEVEL\b/1/is; + } + # Replace call to SYS_CONNECT_BY_PATH by the right concatenation string + if ($columns[$i] =~ s/SYS_CONNECT_BY_PATH\s*[\(]*\s*([^,]+),\s*([^\)]+)\s*\)/$1/is) { + my $col = $1; + $connect_by_path{$col}{sep} = $2; + # get the column alias + if ($columns[$i] =~ /\s+([^\s]+)\s*$/s) { + $connect_by_path{$col}{alias} = $1; + } + } + if ($columns[$i] =~ /([^\.]+)\./s) { + push(@tabalias, $1) if (!grep(/^\Q$1\E$/i, @tabalias)); + } + extract_subpart($class, \$columns[$i]); + + # Append parenthesis on new subqueries values + foreach my $z (sort {$a <=> $b } keys %{$class->{sub_parts}}) { + next if ($class->{sub_parts}{$z} =~ /^\(/); + # If subpart is not empty after transformation + if ($class->{sub_parts}{$z} =~ /\S/is) { + # add open and closed parenthesis + $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; + } elsif ($statements[$i] !~ /\s+(WHERE|AND|OR)\s*\%SUBQUERY$z\%/is) { + # otherwise do not report the empty parenthesis when this is not a function + $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; + } + } + } + + # Extraction of the table aliases in the FROM clause + my $cols = join(',', @columns); + $str =~ s/COLUMN_ALIAS/$cols/s; + if ($str =~ s/(\s+FROM\s+)(.*)/$1FROM_CLAUSE/is) { + my $from_clause = $2; + $str =~ s/FROM_CLAUSE/$from_clause/; + } + + # Now append the UNION ALL query that will be called recursively + $final_query .= $str; + $final_query .= ' WHERE ' . $start_with . "\n" if ($start_with); + #$where_clause =~ s/^\s*WHERE\s+/ AND /is; + #$final_query .= $where_clause . "\n"; + $final_query .= " UNION ALL\n"; + if ($siblings && !$order_by) { + $final_query =~ s/(\s+FROM\s+)/,ARRAY[ row_number() OVER (ORDER BY $siblings) ] as hierarchy$1/is; + } elsif ($siblings) { + + $final_query =~ s/(\s+FROM\s+)/,ARRAY[ row_number() OVER (ORDER BY $order_by) ] as hierarchy$1/is; + } + $bkup_query =~ s/(SELECT\s+)(.*?)(\s+FROM)/$1COLUMN_ALIAS$3/is; + @columns = split(/\s*,\s*/, $2); + # When the pseudo column LEVEL is used in the where clause + # and not used in columns list, add the pseudo column + if ($where_clause =~ /\bLEVEL\b/is && !grep(/\bLEVEL\b/i, @columns)) { + push(@columns, 'level'); + } + for (my $i = 0; $i <= $#columns; $i++) { + my $found = 0; + while ($columns[$i] =~ s/\%SUBQUERY(\d+)\%/$class->{sub_parts}{$1}/is) { + # Get out of here when a call to SYS_CONNECT_BY_PATH is found + # This will prevent opening subquery in the function parameters + last if ($found); + $found = 1 if ($columns[$i]=~ /SYS_CONNECT_BY_PATH/is); + }; + if ($columns[$i] =~ s/SYS_CONNECT_BY_PATH\s*[\(]*\s*([^,]+),\s*([^\)]+)\s*\)/$1/is) { + $columns[$i] = "c.$connect_by_path{$1}{alias} || $connect_by_path{$1}{sep} || " . $columns[$i]; + } + if ($columns[$i] !~ s/\b[^\.]+\.LEVEL\b/(c.level+1)/igs) { + $columns[$i] =~ s/\bLEVEL\b/(c.level+1)/igs; + } + extract_subpart($class, \$columns[$i]); + + # Append parenthesis on new subqueries values + foreach my $z (sort {$a <=> $b } keys %{$class->{sub_parts}}) { + next if ($class->{sub_parts}{$z} =~ /^\(/); + # If subpart is not empty after transformation + if ($class->{sub_parts}{$z} =~ /\S/is) { + # add open and closed parenthesis + $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; + } elsif ($statements[$i] !~ /\s+(WHERE|AND|OR)\s*\%SUBQUERY$z\%/is) { + # otherwise do not report the empty parenthesis when this is not a function + $class->{sub_parts}{$z} = '(' . $class->{sub_parts}{$z} . ')'; + } + } + } + $cols = join(',', @columns); + $bkup_query =~ s/COLUMN_ALIAS/$cols/s; + my $prior_alias = ''; + if ($bkup_query =~ s/(\s+FROM\s+)(.*)/$1FROM_CLAUSE/is) { + my $from_clause = $2; + if ($from_clause =~ /\b[^\s]+\s+(?:AS\s+)?([^\s]+)\b/) { + my $a = $1; + $prior_alias = "$a." if (!grep(/\b$a\.[^\s]+$/, @prior_clause)); + } + $bkup_query =~ s/FROM_CLAUSE/$from_clause/; + } + + # Remove last subquery alias in the from clause to put our own + $bkup_query =~ s/(\%SUBQUERY\d+\%)\s+[^\s]+\s*$/$1/is; + if ($siblings && $order_by) { + $bkup_query =~ s/(\s+FROM\s+)/, array_append(c.hierarchy, row_number() OVER (ORDER BY $order_by)) as hierarchy$1/is; + } elsif ($siblings) { + $bkup_query =~ s/(\s+FROM\s+)/, array_append(c.hierarchy, row_number() OVER (ORDER BY $siblings)) as hierarchy$1/is; + } + $final_query .= $bkup_query; + map { s/^\s*(.*?)(=\s*)(.*)/c\.$1$2$prior_alias$3/s; } @prior_clause; + $final_query .= " JOIN cte c ON (" . join(' AND ', @prior_clause) . ")\n"; + if ($siblings) { + $order_by = " ORDER BY hierarchy"; + } elsif ($order_by) { + $order_by =~ s/^, //s; + $order_by = " ORDER BY $order_by"; + } + $final_query .= "\n) SELECT * FROM cte$where_clause$union$group_by$order_by;\n"; + + return $final_query; +} + +sub replace_without_function +{ + my ($class, $str) = @_; + + # Code disabled because it break other complex GROUP BY clauses + # Keeping it just in case some light help me to solve this problem + # Reported in issue #496 + # Remove text constant in GROUP BY clause, this is not allowed + # GROUP BY ?TEXTVALUE10?, %%REPLACEFCT1%%, DDI.LEGAL_ENTITY_ID + #if ($str =~ s/(\s+GROUP\s+BY\s+)(.*?)((?:(?=\bUNION\b|\bORDER\s+BY\b|\bLIMIT\b|\bINTO\s+|\bFOR\s+UPDATE\b|\bPROCEDURE\b).)+|$)/$1\%GROUPBY\% $3/is) { + # my $tmp = $2; + # $tmp =~ s/\?TEXTVALUE\d+\?[,]*\s*//gs; + # $tmp =~ s/(\s*,\s*),\s*/$1/gs; + # $tmp =~ s/\s*,\s*$//s; + # $str =~ s/\%GROUPBY\%/$tmp/s; + #} + + return $str; +} + +1; + +__END__ + + +=head1 AUTHOR + +Gilles Darold + + +=head1 COPYRIGHT + +Copyright (c) 2000-2020 Gilles Darold - All rights reserved. + +This program is free software; you can redistribute it and/or modify it under +the same terms as Perl itself. + + +=head1 BUGS + +This perl module is in the same state as my knowledge regarding database, +it can move and not be compatible with older version so I will do my best +to give you official support for Ora2Pg. Your volontee to help construct +it and your contribution are welcome. + + +=head1 SEE ALSO + +L + +=cut + diff --git a/packaging/README b/packaging/README new file mode 100644 index 0000000000000000000000000000000000000000..5b195d1918ceb2322a26637eb478b42fef031c90 --- /dev/null +++ b/packaging/README @@ -0,0 +1,56 @@ +RPM/ + Holds ora2pg.spec need to build an RPM package for RH/Fedora. + It may also be usable for other RPM based distribution. + + Copy the ora2pg source tarball here: + + ~/rpmbuild/SOURCES/ or /usr/src/redhat/SOURCES/ + + Then create the RPM binary package as follow: + + rpmbuild -bb ora2pg.spec + + The binary package may be found here: + + ~/rpmbuild/RPMS/noarch/ora2pg-21.1-1.noarch.rpm + or + /usr/src/redhat/RPMS/i386/ora2pg-21.1-1.noarch.rpm + + To install run: + + rpm -i ~/rpmbuild/RPMS/noarch/ora2pg-21.1-1.noarch.rpm + + +slackbuild/ + Holds all files necessary to build a Slackware package. + Copy the source tarball into the slackbuild directory and run + + sh Ora2Pg.SlackBuild + + then take a look at /tmp/build/ to find the Slackware package. + To install run the following command: + + installpkg /tmp/build/ora2pg-21.1-i386-1gda.tgz + + or + + installpkg /tmp/build/ora2pg-21.1-x86_64-1gda.tgz + + following the architecture. + + +debian/ + Holds all files to build debian package. + First you need to execute script 'sh create-deb-tree.sh' in the debian + directory to create the package tree. After that just run the following + command to generate the debian package: + + dpkg -b ora2pg ora2pg.deb + + To install the package, run: + + dpkg -i ora2pg.deb + + +Feel free to send me other. + diff --git a/packaging/RPM/ora2pg.spec b/packaging/RPM/ora2pg.spec new file mode 100644 index 0000000000000000000000000000000000000000..337812d0cf5a76f22c7a62cbbade96d810960314 --- /dev/null +++ b/packaging/RPM/ora2pg.spec @@ -0,0 +1,102 @@ +Summary: Oracle to PostgreSQL database schema converter +Name: ora2pg +Version: 18.0 +Release: 1%{?dist} +Group: Applications/Databases +License: GPLv3+ +URL: http://ora2pg.darold.net/ +Source0: https://github.com/darold/%{name}/archive/v%{version}.tar.gz +BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) +BuildArch: noarch + +BuildRequires: perl +Requires: perl(DBD::Oracle) +Requires: perl-DBD-MySQL perl(DBI) perl(IO::Compress::Base) + +%description +This package contains a Perl module and a companion script to convert an +Oracle database schema to PostgreSQL and to migrate the data from an +Oracle database to a PostgreSQL database. + +%prep +%setup -q + +%build +# Make Perl and Ora2Pg distrib files +%{__perl} Makefile.PL \ + INSTALLDIRS=vendor \ + QUIET=1 \ + CONFDIR=%{_sysconfdir} \ + DOCDIR=%{_docdir}/%{name}-%{version} \ + DESTDIR=%{buildroot} +%{__make} + +%install +%{__rm} -rf %{buildroot} +%{__make} install DESTDIR=%{buildroot} + +# Remove unpackaged files. +%{__rm} -f `find %{buildroot}/%{_libdir}/perl*/ -name .packlist -type f` +%{__rm} -f `find %{buildroot}/%{_libdir}/perl*/ -name perllocal.pod -type f` + + +%clean +%{__rm} -rf %{buildroot} + +%files +%defattr(-, root, root, 0755) +%attr(0755,root,root) %{_bindir}/%{name} +%attr(0755,root,root) %{_bindir}/%{name}_scanner +%attr(0644,root,root) %{_mandir}/man3/%{name}.3.gz +%config(noreplace) %{_sysconfdir}/%{name}.conf.dist +#%config(noreplace) %{_sysconfdir}/%{name}/%{name}.conf.dist +%{perl_vendorlib}/Ora2Pg/MySQL.pm +%{perl_vendorlib}/Ora2Pg/PLSQL.pm +%{perl_vendorlib}/Ora2Pg/GEOM.pm +%{perl_vendorlib}/Ora2Pg.pm +%{_docdir}/%{name}-%{version}/* + +%changelog +* Tue Jan 31 2017 Devrim Gündüz 18.0-1 +- Update to 18.0 + +* Mon Nov 21 2016 Devrim Gündüz 17.6-1 +- Update to 17.6 + +* Fri Oct 21 2016 Devrim Gündüz 17.5-1 +- Update to 17.5 + +* Mon Apr 18 2016 Devrim Gündüz 17.3-1 +- Update to 17.3 + +* Fri Mar 25 2016 Devrim Gündüz 17.2-1 +- Update to 17.2 + +* Wed Mar 9 2016 Devrim Gündüz 17.1-1 +- Update to 17.1 + +* Thu Jan 21 2016 Devrim Gündüz 16.2-1 +- Update to 16.2 + +* Wed Dec 30 2015 Devrim GUNDUZ 16.1-1 +- Update to 16.1 + +* Fri Feb 6 2015 Devrim GUNDUZ 15.1-1 +- Update to 15.1, per changes described at: + http://www.postgresql.org/message-id/54D49C0B.2000006@dalibo.com + +* Wed Oct 23 2013 Devrim GUNDUZ 12.0-1 +- Update to 12.0, per changes described at: + http://www.postgresql.org/message-id/52664854.30200@dalibo.com + +* Thu Sep 12 2013 Devrim GUNDUZ 11.4-1 +- Update to 11.4 + +* Thu Sep 13 2012 Devrim GUNDUZ 9.2-1 +- Update to 9.2 +- Update URL, License, Group tags +- Fix spec per rpmlint warnings +- Apply some changes from upstream spec + +* Fri Mar 20 2009 Devrim GUNDUZ 5.0-1 +- Initial release, based on Peter's spec file. diff --git a/packaging/debian/create-deb-tree.sh b/packaging/debian/create-deb-tree.sh new file mode 100644 index 0000000000000000000000000000000000000000..565c68a3493b253c13621372e7136a2a41c49915 --- /dev/null +++ b/packaging/debian/create-deb-tree.sh @@ -0,0 +1,23 @@ +#!/bin/sh +# +# Script used to create the Debian package tree. This script must be +# executed in his directory +# +LPWD=`pwd` +DEST=packaging/debian/ora2pg +cd ../../ +perl Makefile.PL \ + INSTALLDIRS=vendor \ + QUIET=1 \ + CONFDIR=/etc/ora2pg \ + DOCDIR=/usr/share/doc/ora2pg \ + DESTDIR=$DEST || exit 1 + +make && make install DESTDIR=$DEST + +echo "Compressing man pages" +find $DEST/usr/share/man/ -type f -name "*.?" -exec gzip -9 {} \; +find $DEST/usr/share/man/ -type f -name "*.?pm" -exec gzip -9 {} \; + +cd $LPWD + diff --git a/packaging/debian/ora2pg/DEBIAN/control b/packaging/debian/ora2pg/DEBIAN/control new file mode 100644 index 0000000000000000000000000000000000000000..1164c0000156d9d58f5b4ad8c705ab1c574682ed --- /dev/null +++ b/packaging/debian/ora2pg/DEBIAN/control @@ -0,0 +1,15 @@ +Package: ora2pg +Version: 21.1 +Priority: optional +Architecture: all +Essential: no +Depends: libcompress-zlib-perl, libdbd-oracle-perl, libdbi-perl, libdbd-mysql-perl +Pre-Depends: perl +Recommends: postgresql-client +Installed-Size: 1024 +Maintainer: Gilles Darold +Provide: ora2pg +Description: Ora2Pg (Oracle to PostgreSQL database schema converter) + This package contains all Perl modules and scripts to convert an + Oracle or MySQL database schema, data and stored procedures to a + PostgreSQL database. diff --git a/packaging/debian/ora2pg/DEBIAN/copyright b/packaging/debian/ora2pg/DEBIAN/copyright new file mode 100644 index 0000000000000000000000000000000000000000..3a88f4333b47b1fee734dd09216aad232d320565 --- /dev/null +++ b/packaging/debian/ora2pg/DEBIAN/copyright @@ -0,0 +1,43 @@ +This work was packaged for Debian by: + + Peter Eisentraut on 2007-10-24 + Julin Moreno Patio on Sun, 21 Mar 2010 07:18:35 -0500 + +It was downloaded from: + + http://ora2pg.darold.net/ + +Upstream Author(s): + + Gilles Darold + +Copyright: + + Copyright (c) 2000-2020 : Gilles Darold - All rights reserved + +License: + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This package is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +On Debian systems, the complete text of the GNU General +Public License version 3 can be found in "/usr/share/common-licenses/GPL-3". + +The Debian packaging is: + + Copyright (C) 2007 Peter Eisentraut + Copyright (C) 2009 Julin Moreno Patio + +and is licensed under the GPL version 2, +see `/usr/share/common-licenses/GPL-2'. + diff --git a/packaging/slackbuild/Ora2Pg.SlackBuild b/packaging/slackbuild/Ora2Pg.SlackBuild new file mode 100644 index 0000000000000000000000000000000000000000..6dc0afc9a2abdca4dee0f921396a007a31a56c10 --- /dev/null +++ b/packaging/slackbuild/Ora2Pg.SlackBuild @@ -0,0 +1,155 @@ +#!/bin/sh +# Written by Gilles Darold < gilles at darold dot net > +# Licence: GPL v3 +# +# Build script for Slackware - Ora2Pg SlackBuild +# +# Latest Software sourcecode is available at: +# http://ora2pg.darold.net/ +# +# Depends: d/perl +# Suggests: + +## Fill these variables to your needs ## +NAMESRC=${NAMESRC:-ora2pg} +VERSION=${VERSION:-21.1} +EXT=${EXT:-tar.bz2} +NAMEPKG=${NAMEPKG:-ora2pg} +PKGEXT=${PKGEXT:-tgz/txz} +BUILD=${BUILD:-1} +TAG=${TAG:-_gda} +PNAME=${PNAME:-ora2pg} + +TMP=${TMP:-/tmp} +OUT=${OUT:-$TMP/build} +ARCH=${ARCH:-i486} +TARGET=${TARGET:-i486} +WGET=${WGET:-http://downloads.sourceforge.net/ora2pg/$NAMESRC-$VERSION.$EXT} +DOC="Change* INSTALL README" +SUM="" +######################################## + +set -e +umask 022 + +if [ ! -r $NAMESRC-$VERSION.$EXT ]; then + wget -vc $WGET -O $NAMESRC-$VERSION.$EXT.part + mv $NAMESRC-$VERSION.$EXT.part $NAMESRC-$VERSION.$EXT +fi + +# if checksum is include in the script : generate and check +if [ -n "$SUM" ]; then +echo "$SUM $NAMESRC-$VERSION.$EXT" > $NAMESRC-$VERSION.$EXT.sha1 +sha1sum -c $NAMESRC-$VERSION.$EXT.sha1 +elif [ -f $NAMESRC-$VERSION.$EXT.sha1 ]; then +sha1sum -c $NAMESRC-$VERSION.$EXT.sha1 +fi + +# or just check if the .sha1 is another file + +CWD=$(pwd) +PKG=$TMP/build/$NAMEPKG +NAME=$(tar ft $NAMESRC-$VERSION.$EXT | head -n 1 | awk -F/ '{ print $1 }') + +case $ARCH in + i386)SLKCFLAGS="-O2 -march=i386 -mtune=i686";LIBDIRSUFFIX="";; + i486)SLKCFLAGS="-O2 -march=i486 -mtune=i686";LIBDIRSUFFIX="";; + i586)SLKCFLAGS="-O2 -march=i586 -mtune=i686";LIBDIRSUFFIX="";; + i686)SLKCFLAGS="-O2 -march=i686 -mtune=i686";LIBDIRSUFFIX="";; + s390)SLKCFLAGS="-O2";LIBDIRSUFFIX="";; + x86_64)SLKCFLAGS="-O2 -fPIC";LIBDIRSUFFIX="64" + esac + +if [ "$(id -u)" = "0" ]; then + echo "You shouldn't run this SlackBuild as ROOT !" + exit 1 +fi + +if [ ! -d $TMP ]; then + echo "$TMP doesn't exist or is not a directory !" + exit 1 +fi + +# Build the software +cd $TMP + +echo "Building $NAMESRC-$VERSION.$EXT..." +tar xf $CWD/$NAMESRC-$VERSION.$EXT +cd $NAME +perl Makefile.PL \ + INSTALLDIRS=vendor \ + QUIET=1 \ + CONFDIR=/etc/$PNAME \ + DOCDIR=/usr/share/doc/$PNAME \ + DESTDIR=$PKG || exit 1 + +make +make install DESTDIR=$PKG +# Please note that some software use INSTALL_ROOT=$PKG or prefix=$PKG/usr or install_root=$PKG ... + +# Install a slack-desc +mkdir -p $PKG/install +cat $CWD/slack-desc > $PKG/install/slack-desc + +# Install a doinst.sh, if it exists +if [ -r $CWD/doinst.sh ]; then + cat $CWD/doinst.sh > $PKG/install/doinst.sh +fi + +mkdir -p $PKG/usr/doc/$PNAME-$VERSION +cp -a $DOC $PKG/usr/doc/$PNAME-$VERSION + +# Compress the man pages +if [ -d $PKG/usr/man ]; then + find $PKG/usr/man -type f -name "*.?" -exec gzip -9 {} \; + for manpage in $(find $PKG/usr/man -type l) ; do + ln -s $(readlink $manpage).gz $manpage.gz + rm -f $manpage + done +fi +if [ -d $PKG/usr/share/man ]; then + find $PKG/usr/share/man -type f -name "*.?" -exec gzip -9 {} \; + for manpage in $(find $PKG/usr/share/man -type l) ; do + ln -s $(readlink $manpage).gz $manpage.gz + rm -f $manpage + done +fi + +# Compress the info pages +if [ -d $PKG/usr/info ]; then + rm -f $PKG/usr/info/dir + gzip -9 $PKG/usr/info/*.info* +fi + +# Remove 'special' files +find $PKG -name perllocal.pod \ +-o -name ".packlist" \ +-o -name "*.bs" \ +| xargs rm -f +# Remove empty directory +rmdir --parents $PKG/usr/lib/perl5/5.*/i486-linux-thread-multi 2>/dev/null || true +rmdir --parents $PKG/usr/lib/perl5/vendor_perl/5.*/i486-linux-thread-multi/auto/Ora2Pg 2>/dev/null || true + +# Strip binaries, libraries and archives +find $PKG -type f | xargs file | grep "LSB executable" | cut -f 1 -d : | xargs \ + strip --strip-unneeded 2> /dev/null || echo "No binaries to strip" +find $PKG -type f | xargs file | grep "shared object" | cut -f 1 -d : | xargs \ + strip --strip-unneeded 2> /dev/null || echo "No shared objects to strip" +find $PKG -type f | xargs file | grep "current ar archive" | cut -f 1 -d : | \ + xargs strip -g 2> /dev/null || echo "No archives to strip" + +# Build the package +cd $PKG +mkdir -p $OUT +PACKAGING=" +chown root:root . -R +/sbin/makepkg -l y -c n $OUT/$NAMEPKG-$VERSION-$ARCH-$BUILD$TAG.tgz +rm -rf $PKG +rm -rf $TMP/$NAME +" +if [ "$(which fakeroot 2> /dev/null)" ]; then + echo "$PACKAGING" | fakeroot +else + su -c "$PACKAGING" +fi + diff --git a/packaging/slackbuild/Ora2Pg.info b/packaging/slackbuild/Ora2Pg.info new file mode 100644 index 0000000000000000000000000000000000000000..894feacb8f5ce20d7e5265e4f01d507b6ce0d60b --- /dev/null +++ b/packaging/slackbuild/Ora2Pg.info @@ -0,0 +1,10 @@ +PRGNAM="Ora2Pg" +VERSION="21.1" +HOMEPAGE="http://ora2pg.darold.net/" +DOWNLOAD="http://downloads.sourceforge.net/ora2pg/ora2pg-21.1.tar.gz" +MD5SUM="" +DOWNLOAD_x86_64="UNTESTED" +MD5SUM_x86_64="" +MAINTAINER="Gilles Darold" +EMAIL="gilles@darold.net" +APPROVED="" diff --git a/packaging/slackbuild/README b/packaging/slackbuild/README new file mode 100644 index 0000000000000000000000000000000000000000..c28b81ffbdd13eed895e8d8d1ac8cb259518dfa1 --- /dev/null +++ b/packaging/slackbuild/README @@ -0,0 +1,21 @@ +Ora2Pg is a tool used to migrate an Oracle database to a PostgreSQL compatible +schema. It connects your Oracle database, scan it automaticaly and extracts its +structure or data, it then generates SQL scripts that you can load into your +PostgreSQL database. + +Ora2Pg can be used from reverse engineering Oracle database to huge enterprise +database migration or simply to replicate some Oracle data into a PostgreSQL +database. It is really easy to used and doesn't need any Oracle database +knowledge than providing the parameters needed to connect to the Oracle +database. + +You need a modern Perl distribution (perl 5.6 or more), the DBI and DBD::Oracle +Perl modules to be installed. These are used to connect to the Oracle database. +To install DBD::Oracle and have it working you need to have the Oracle client +libraries installed and the ORACLE_HOME environment variable must be defined. + +Note that the Oracle and the PostgreSQL databases doesn't need to be on the +host running Ora2Pg but this host must have at least the Oracle client libraries +installed and the PostgreSQL client if you want to use psql or DBD::Pg to import +data. + diff --git a/packaging/slackbuild/doinst.sh b/packaging/slackbuild/doinst.sh new file mode 100644 index 0000000000000000000000000000000000000000..f9c723b23a0696630a5f320ca9ab08855955fadd --- /dev/null +++ b/packaging/slackbuild/doinst.sh @@ -0,0 +1,4 @@ +#! /bin/sh + +cat README + diff --git a/packaging/slackbuild/slack-desc b/packaging/slackbuild/slack-desc new file mode 100644 index 0000000000000000000000000000000000000000..cee7f936a74bbd31dda239876d015e95df603d2b --- /dev/null +++ b/packaging/slackbuild/slack-desc @@ -0,0 +1,12 @@ + |-----handy-ruler------------------------------------------------------| +Ora2Pg: Ora2Pg (Oracle to PostgreSQL database schema converter) +Ora2Pg: +Ora2Pg: This package contains Perl modules and a companion script to convert an +Ora2Pg: Oracle database schema to PostgreSQL and to migrate the data from an +Ora2Pg: Oracle database to a PostgreSQL database. +Ora2Pg: +Ora2Pg: Ora2Pg can be used from reverse engineering Oracle database to huge +Ora2Pg: enterprise database migration or simply to replicate some Oracle data +Ora2Pg: into a PostgreSQL database. +Ora2Pg: +Ora2Pg: diff --git a/scripts/ora2pg b/scripts/ora2pg new file mode 100644 index 0000000000000000000000000000000000000000..7b365685b5efac785eec6c54a7c7d768717fad5d --- /dev/null +++ b/scripts/ora2pg @@ -0,0 +1,1092 @@ +#!/usr/bin/perl +#------------------------------------------------------------------------------ +# Project : Oracle to Postgresql converter +# Name : ora2pg +# Author : Gilles Darold, gilles _AT_ darold _DOT_ net +# Copyright: Copyright (c) 2000-2020 : Gilles Darold - All rights reserved - +# Function : Script used to convert Oracle Database to PostgreSQL +# Usage : ora2pg configuration_file +#------------------------------------------------------------------------------ +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see < http://www.gnu.org/licenses/ >. +# +#------------------------------------------------------------------------------ +use strict qw/vars/; + +use Ora2Pg; +use Getopt::Long qw(:config no_ignore_case bundling); +use File::Spec qw/ tmpdir /; +use POSIX qw(locale_h sys_wait_h _exit); +setlocale(LC_NUMERIC, ''); +setlocale(LC_ALL, 'C'); + +my $VERSION = '21.1'; + +$| = 1; + +my $CONFIG_FILE = "/etc/ora2pg/ora2pg.conf"; +my $FILE_CONF = ''; +my $DEBUG = 0; +my $QUIET = 0; +my $HELP = 0; +my $LOGFILE = ''; +my $EXPORT_TYPE = ''; +my $OUTFILE = ''; +my $OUTDIR = ''; +my $SHOW_VER = 0; +my $PLSQL = ''; +my $DSN = ''; +my $DBUSER = ''; +my $DBPWD = ''; +my $SCHEMA = ''; +my $TABLEONLY = ''; +my $FORCEOWNER = ''; +my $ORA_ENCODING = ''; +my $PG_ENCODING = ''; +my $INPUT_FILE = ''; +my $EXCLUDE = ''; +my $ALLOW = ''; +my $VIEW_AS_TABLE = ''; +my $ESTIMATE_COST; +my $COST_UNIT_VALUE; +my $DUMP_AS_HTML; +my $DUMP_AS_CSV; +my $DUMP_AS_SHEET; +my $THREAD_COUNT; +my $ORACLE_COPIES; +my $PARALLEL_TABLES; +my $DATA_LIMIT; +my $CREATE_PROJECT = ''; +my $PROJECT_BASE = '.'; +my $PRINT_HEADER = ''; +my $HUMAN_DAY_LIMIT; +my $IS_MYSQL = 0; +my $AUDIT_USER = ''; +my $PG_DSN = ''; +my $PG_USER = ''; +my $PG_PWD = ''; +my $COUNT_ROWS = 0; +my $DATA_TYPE = ''; +my $GRANT_OBJECT = ''; +my $PG_SCHEMA = ''; +my $NO_HEADER = 0; +my $ORACLE_SPEED = 0; +my $ORA2PG_SPEED = 0; +my $RELATIVE_PATH = 0; + +my @SCHEMA_ARRAY = qw( SEQUENCE TABLE PACKAGE VIEW GRANT TRIGGER FUNCTION PROCEDURE TABLESPACE PARTITION TYPE MVIEW DBLINK SYNONYM DIRECTORY ); +my @EXTERNAL_ARRAY = qw( KETTLE FDW ); +my @REPORT_ARRAY = qw( SHOW_VERSION SHOW_REPORT SHOW_SCHEMA SHOW_TABLE SHOW_COLUMN SHOW_ENCODING ); +my @TEST_ARRAY = qw( TEST TEST_VIEW); +my @SOURCES_ARRAY = qw( PACKAGE VIEW TRIGGER FUNCTION PROCEDURE PARTITION TYPE MVIEW ); +my @DATA_ARRAY = qw( INSERT COPY ); +my @CAPABILITIES = qw( QUERY LOAD ); + +my @MYSQL_SCHEMA_ARRAY = qw( TABLE VIEW GRANT TRIGGER FUNCTION PROCEDURE PARTITION DBLINK ); +my @MYSQL_SOURCES_ARRAY = qw( VIEW TRIGGER FUNCTION PROCEDURE PARTITION ); + +my @GRANT_OBJECTS_ARRAY = ('USER','TABLE','VIEW','MATERIALIZED VIEW','SEQUENCE','PROCEDURE','FUNCTION','PACKAGE BODY','TYPE','SYNONYM','DIRECTORY'); + +my $TMP_DIR = File::Spec->tmpdir() || '/tmp'; + +# Collect command line arguments +GetOptions ( + 'a|allow=s' => \$ALLOW, + 'b|basedir=s' => \$OUTDIR, + 'c|conf=s' => \$FILE_CONF, + 'd|debug!' => \$DEBUG, + 'D|data_type=s' => \$DATA_TYPE, + 'e|exclude=s' => \$EXCLUDE, + 'g|grant_object=s' => \$GRANT_OBJECT, + 'h|help!' => \$HELP, + 'i|input_file=s' => \$INPUT_FILE, + 'j|jobs=i' => \$THREAD_COUNT, + 'J|copies=i' => \$ORACLE_COPIES, + 'l|log=s' => \$LOGFILE, + 'L|limit=i' => \$DATA_LIMIT, + 'm|mysql!' => \$IS_MYSQL, + 'n|namespace=s' => \$SCHEMA, + 'N|pg_schema=s' => \$PG_SCHEMA, + 'o|out=s' => \$OUTFILE, + 'p|plsql!' => \$PLSQL, + 'P|parallel=i' =>\$PARALLEL_TABLES, + 'q|quiet!' => \$QUIET, + 'r|relative!' => \$RELATIVE_PATH, + 's|source=s' => \$DSN, + 't|type=s' => \$EXPORT_TYPE, + 'T|temp_dir=s' => \$TMP_DIR, + 'u|user=s' => \$DBUSER, + 'v|version!' => \$SHOW_VER, + 'w|password=s' => \$DBPWD, + 'x|xtable=s' => \$TABLEONLY, # Obsolete + 'forceowner=s' => \$FORCEOWNER, + 'nls_lang=s' => \$ORA_ENCODING, + 'client_encoding=s' => \$PG_ENCODING, + 'view_as_table=s' => \$VIEW_AS_TABLE, + 'estimate_cost!' =>\$ESTIMATE_COST, + 'cost_unit_value=i' =>\$COST_UNIT_VALUE, + 'dump_as_html!' =>\$DUMP_AS_HTML, + 'dump_as_csv!' =>\$DUMP_AS_CSV, + 'dump_as_sheet!' =>\$DUMP_AS_SHEET, + 'init_project=s' => \$CREATE_PROJECT, + 'project_base=s' => \$PROJECT_BASE, + 'print_header!' => \$PRINT_HEADER, + 'human_days_limit=i' => \$HUMAN_DAY_LIMIT, + 'audit_user=s' => \$AUDIT_USER, + 'pg_dsn=s' => \$PG_DSN, + 'pg_user=s' => \$PG_USER, + 'pg_pwd=s' => \$PG_PWD, + 'count_rows!' => \$COUNT_ROWS, + 'no_header!' => \$NO_HEADER, + 'oracle_speed!' => \$ORACLE_SPEED, + 'ora2pg_speed!' => \$ORA2PG_SPEED, +); + +# Check command line parameters +if ($SHOW_VER) { + print "Ora2Pg v$VERSION\n"; + exit 0; +} +if ($HELP) { + &usage(); +} + +if ($IS_MYSQL) { + @SCHEMA_ARRAY = @MYSQL_SCHEMA_ARRAY; + @SOURCES_ARRAY = @MYSQL_SOURCES_ARRAY; + @EXTERNAL_ARRAY = (); +} + +# Create project repository and useful stuff +if ($CREATE_PROJECT) { + if (!-d "$PROJECT_BASE") { + print "FATAL: Project base directory does not exists: $PROJECT_BASE\n"; + &usage(); + } + print STDERR "Creating project $CREATE_PROJECT.\n"; + &create_project($CREATE_PROJECT, $PROJECT_BASE); + exit 0; +} + +if ($GRANT_OBJECT && !grep(/^$GRANT_OBJECT$/, @GRANT_OBJECTS_ARRAY)) { + print "FATAL: invalid grant object type in -g option. See GRAND_OBJECT configuration directive.\n"; + exit 1; +} + +# Clean temporary files +unless(opendir(DIR, "$TMP_DIR")) { + print "FATAL: can't opendir $TMP_DIR: $!\n"; + exit 1; +} +my @files = grep { $_ =~ /^tmp_ora2pg.*$/ } readdir(DIR); +closedir DIR; +foreach (@files) { + if (not unlink("$TMP_DIR/$_\n")){ + print "FATAL: can not remove old temporary files $TMP_DIR/$_\n"; + exit 1; + } +} + +# Check configuration file +my $GOES_WITH_DEFAULT = 0; +if ($FILE_CONF && ! -e $FILE_CONF) { + print "FATAL: can't find configuration file $FILE_CONF\n"; + &usage(); +} elsif (!$FILE_CONF && ! -e $CONFIG_FILE) { + # At least we need configuration to connect to Oracle + if (!$DSN || (!$DBUSER && !$ENV{ORA2PG_USER}) || (!$DBPWD && !$ENV{ORA2PG_PASSWD})) { + print "FATAL: can't find configuration file $CONFIG_FILE\n"; + &usage(); + } + $CONFIG_FILE = ''; + $GOES_WITH_DEFAULT = 1; +} + +push(@CAPABILITIES, @SCHEMA_ARRAY, @REPORT_ARRAY, @DATA_ARRAY, @EXTERNAL_ARRAY, @TEST_ARRAY); + +# Validate export type +$EXPORT_TYPE = uc($EXPORT_TYPE); +$EXPORT_TYPE =~ s/DATA/COPY/; +foreach my $t (split(/[,;\s\t]+/, $EXPORT_TYPE)) { + if ($t && !grep(/^$t$/, @CAPABILITIES)) { + print "FATAL: Unknown export type: $t. Type supported: ", join(',', @CAPABILITIES), "\n"; + &usage(); + } +} + +# Preserve barckward compatibility +if ($TABLEONLY) { + warn "-x | --xtable is deprecated, use -a | --allow option instead.\n"; + if (!$ALLOW) { + $ALLOW = $TABLEONLY; + } +} + +sub getout +{ + my $sig = shift; + print STDERR "Received terminating signal ($sig).\n"; + $SIG{INT} = \&getout; + $SIG{TERM} = \&getout; + + # Cleaning temporary files + unless(opendir(DIR, "$TMP_DIR")) { + print "FATAL: can't opendir $TMP_DIR: $!\n"; + exit 1; + } + my @files = grep { $_ =~ /^tmp_ora2pg.*$/ } readdir(DIR); + closedir DIR; + foreach (@files) { + unlink("$TMP_DIR/$_\n"); + } + + exit 1; +} +$SIG{INT} = \&getout; +$SIG{TERM} = \&getout; + +# Replace ; or space by comma in the user list +$AUDIT_USER =~ s/[;\s]+/,/g; + +# Create an instance of the Ora2Pg perl module +my $schema = new Ora2Pg ( + config => $FILE_CONF || $CONFIG_FILE, + type => $EXPORT_TYPE, + debug => $DEBUG, + logfile=> $LOGFILE, + output => $OUTFILE, + output_dir => $OUTDIR, + plsql_pgsql => $PLSQL, + datasource => $DSN, + user => $DBUSER || $ENV{ORA2PG_USER}, + password => $DBPWD || $ENV{ORA2PG_PASSWD}, + schema => $SCHEMA, + pg_schema => $PG_SCHEMA, + force_owner => $FORCEOWNER, + nls_lang => $ORA_ENCODING, + client_encoding => $PG_ENCODING, + input_file => $INPUT_FILE, + quiet => $QUIET, + exclude => $EXCLUDE, + allow => $ALLOW, + view_as_table => $VIEW_AS_TABLE, + estimate_cost => $ESTIMATE_COST, + cost_unit_value => $COST_UNIT_VALUE, + dump_as_html => $DUMP_AS_HTML, + dump_as_csv => $DUMP_AS_CSV, + dump_as_sheet => $DUMP_AS_SHEET, + thread_count => $THREAD_COUNT, + oracle_copies => $ORACLE_COPIES, + data_limit => $DATA_LIMIT, + parallel_tables => $PARALLEL_TABLES, + print_header => $PRINT_HEADER, + human_days_limit => $HUMAN_DAY_LIMIT, + is_mysql => $IS_MYSQL, + audit_user => $AUDIT_USER, + temp_dir => $TMP_DIR, + pg_dsn => $PG_DSN, + pg_user => $PG_USER, + pg_pwd => $PG_PWD, + count_rows => $COUNT_ROWS, + data_type => $DATA_TYPE, + grant_object => $GRANT_OBJECT, + no_header => $NO_HEADER, + oracle_speed => $ORACLE_SPEED, + ora2pg_speed => $ORA2PG_SPEED, + psql_relative_path => $RELATIVE_PATH, +); + +# Look at configuration file if an input file is defined +if (!$INPUT_FILE && !$GOES_WITH_DEFAULT) { + my $cf_file = $FILE_CONF || $CONFIG_FILE; + my $fh = new IO::File; + $fh->open($cf_file) or die "FATAL: can't read configuration file $cf_file, $!\n"; + while (my $l = <$fh>) { + chomp($l); + $l =~ s/\r//gs; + $l =~ s/^\s*\#.*$//g; + next if (!$l || ($l =~ /^\s+$/)); + $l =~ s/^\s*//; $l =~ s/\s*$//; + my ($var, $val) = split(/\s+/, $l, 2); + $var = uc($var); + if ($var eq 'INPUT_FILE' && $val) { + $INPUT_FILE = $val; + } + } + $fh->close(); +} + +# Proceed to Oracle DB extraction following +# configuration file definitions. +if ( ($EXPORT_TYPE !~ /^SHOW_/i) && !$INPUT_FILE ) { + $schema->export_schema(); +} + +# Check if error occurs during data export +unless(opendir(DIR, "$TMP_DIR")) { + print "FATAL: can't opendir $TMP_DIR: $!\n"; + exit 1; +} +@files = grep { $_ =~ /^tmp_ora2pg.*$/ } readdir(DIR); +closedir DIR; +if ($#files >= 0) { + print STDERR "\nWARNING: an error occurs during data export. Please check what's happen.\n\n"; + exit 2; +} + +exit(0); + +#### +# Show usage +#### +sub usage +{ + print qq{ +Usage: ora2pg [-dhpqv --estimate_cost --dump_as_html] [--option value] + + -a | --allow str : Comma separated list of objects to allow from export. + Can be used with SHOW_COLUMN too. + -b | --basedir dir: Set the default output directory, where files + resulting from exports will be stored. + -c | --conf file : Set an alternate configuration file other than the + default /etc/ora2pg/ora2pg.conf. + -d | --debug : Enable verbose output. + -D | --data_type STR : Allow custom type replacement at command line. + -e | --exclude str: Comma separated list of objects to exclude from export. + Can be used with SHOW_COLUMN too. + -h | --help : Print this short help. + -g | --grant_object type : Extract privilege from the given object type. + See possible values with GRANT_OBJECT configuration. + -i | --input file : File containing Oracle PL/SQL code to convert with + no Oracle database connection initiated. + -j | --jobs num : Number of parallel process to send data to PostgreSQL. + -J | --copies num : Number of parallel connections to extract data from Oracle. + -l | --log file : Set a log file. Default is stdout. + -L | --limit num : Number of tuples extracted from Oracle and stored in + memory before writing, default: 10000. + -m | --mysql : Export a MySQL database instead of an Oracle schema. + -n | --namespace schema : Set the Oracle schema to extract from. + -N | --pg_schema schema : Set PostgreSQL's search_path. + -o | --out file : Set the path to the output file where SQL will + be written. Default: output.sql in running directory. + -p | --plsql : Enable PLSQL to PLPGSQL code conversion. + -P | --parallel num: Number of parallel tables to extract at the same time. + -q | --quiet : Disable progress bar. + -r | --relative : use \\ir instead of \\i in the psql scripts generated. + -s | --source DSN : Allow to set the Oracle DBI datasource. + -t | --type export: Set the export type. It will override the one + given in the configuration file (TYPE). + -T | --temp_dir DIR: Set a distinct temporary directory when two + or more ora2pg are run in parallel. + -u | --user name : Set the Oracle database connection user. + ORA2PG_USER environment variable can be used instead. + -v | --version : Show Ora2Pg Version and exit. + -w | --password pwd : Set the password of the Oracle database user. + ORA2PG_PASSWD environment variable can be used instead. + --forceowner : Force ora2pg to set tables and sequences owner like in + Oracle database. If the value is set to a username this one + will be used as the objects owner. By default it's the user + used to connect to the Pg database that will be the owner. + --nls_lang code: Set the Oracle NLS_LANG client encoding. + --client_encoding code: Set the PostgreSQL client encoding. + --view_as_table str: Comma separated list of views to export as table. + --estimate_cost : Activate the migration cost evaluation with SHOW_REPORT + --cost_unit_value minutes: Number of minutes for a cost evaluation unit. + default: 5 minutes, corresponds to a migration conducted by a + PostgreSQL expert. Set it to 10 if this is your first migration. + --dump_as_html : Force ora2pg to dump report in HTML, used only with + SHOW_REPORT. Default is to dump report as simple text. + --dump_as_csv : As above but force ora2pg to dump report in CSV. + --dump_as_sheet : Report migration assessment with one CSV line per database. + --init_project NAME: Initialise a typical ora2pg project tree. Top directory + will be created under project base dir. + --project_base DIR : Define the base dir for ora2pg project trees. Default + is current directory. + --print_header : Used with --dump_as_sheet to print the CSV header + especially for the first run of ora2pg. + --human_days_limit num : Set the number of human-days limit where the migration + assessment level switch from B to C. Default is set to + 5 human-days. + --audit_user LIST : Comma separated list of usernames to filter queries in + the DBA_AUDIT_TRAIL table. Used only with SHOW_REPORT + and QUERY export type. + --pg_dsn DSN : Set the datasource to PostgreSQL for direct import. + --pg_user name : Set the PostgreSQL user to use. + --pg_pwd password : Set the PostgreSQL password to use. + --count_rows : Force ora2pg to perform a real row count in TEST action. + --no_header : Do not append Ora2Pg header to output file + --oracle_speed : Use to know at which speed Oracle is able to send + data. No data will be processed or written. + --ora2pg_speed : Use to know at which speed Ora2Pg is able to send + transformed data. Nothing will be written. + +See full documentation at https://ora2pg.darold.net/ for more help or see +manpage with 'man ora2pg'. + +ora2pg will return 0 on success, 1 on error. It will return 2 when a child +process has been interrupted and you've gotten the warning message: + "WARNING: an error occurs during data export. Please check what's happen." +Most of the time this is an OOM issue, first try reducing DATA_LIMIT value. + +}; + exit 1; + +} + +#### +# Create a generic project tree +#### +sub create_project +{ + my ($create_project, $project_base) = @_; + + # Look at default configuration file to use + my $conf_file = $CONFIG_FILE . '.dist'; + if ($FILE_CONF) { + # Use file given in parameter + $conf_file = $FILE_CONF; + } + if (!-f $conf_file || -z $conf_file) { + print "FATAL: file $conf_file does not exists.\n"; + exit 1; + } + # Build entire project tree + my $base_path = $project_base . '/' . $create_project; + if (-e $base_path) { + print "FATAL: project directory exists $base_path\n"; + exit 1; + } + mkdir("$base_path"); + print "$base_path/\n"; + mkdir("$base_path/schema"); + print "\tschema/\n"; + + foreach my $exp (sort @SCHEMA_ARRAY ) { + my $tpath = lc($exp); + $tpath =~ s/y$/ie/; + mkdir("$base_path/schema/" . $tpath . 's'); + print "\t\t" . $tpath . "s/\n"; + } + mkdir("$base_path/sources"); + print "\tsources/\n"; + foreach my $exp (sort @SOURCES_ARRAY ) { + my $tpath = lc($exp); + $tpath =~ s/y$/ie/; + mkdir("$base_path/sources/" . $tpath . 's'); + print "\t\t" . $tpath . "s/\n"; + } + mkdir("$base_path/data"); + print "\tdata/\n"; + mkdir("$base_path/config"); + print "\tconfig/\n"; + mkdir("$base_path/reports"); + print "\treports/\n"; + print "\n"; + + # Copy configuration file and transform it as a generic one + print "Generating generic configuration file\n"; + if (open(IN, "$conf_file")) { + my @cf = ; + close(IN); + # Create a generic configuration file only if it has the .dist extension + # otherwise use the configuration given at command line (-c option) + if ($conf_file =~ /\.dist/) { + &make_config_generic(\@cf); + } + unless(open(OUT, ">$base_path/config/ora2pg.conf")) { + print "FATAL: can't write to file $base_path/config/ora2pg.conf\n"; + exit 1; + } + print OUT @cf; + close(OUT); + } else { + print "FATAL: can not read file $conf_file, $!.\n"; + exit 1; + } + + # Generate shell script to execute all export + print "Creating script export_schema.sh to automate all exports.\n"; + unless(open(OUT, "> $base_path/export_schema.sh")) { + print "FATAL: Can't write to file $base_path/export_schema.sh\n"; + exit 1; + } + print OUT qq{#!/bin/sh +#------------------------------------------------------------------------------- +# +# Generated by Ora2Pg, the Oracle database Schema converter, version $VERSION +# +#------------------------------------------------------------------------------- +}; + print OUT "EXPORT_TYPE=\"", join(' ', @SCHEMA_ARRAY), "\"\n"; + print OUT "SOURCE_TYPE=\"", join(' ', @SOURCES_ARRAY), "\"\n"; + print OUT "namespace=\".\"\n"; + print OUT qq{ +ora2pg -t SHOW_TABLE -c \$namespace/config/ora2pg.conf > \$namespace/reports/tables.txt +ora2pg -t SHOW_COLUMN -c \$namespace/config/ora2pg.conf > \$namespace/reports/columns.txt +ora2pg -t SHOW_REPORT -c \$namespace/config/ora2pg.conf --dump_as_html --estimate_cost > \$namespace/reports/report.html + +for etype in \$(echo \$EXPORT_TYPE | tr " " "\\n") +do + ltype=`echo \$etype | tr '[:upper:]' '[:lower:]'` + ltype=`echo \$ltype | sed 's/y\$/ie/'` + echo "Running: ora2pg -p -t \$etype -o \$ltype.sql -b \$namespace/schema/\$\{ltype\}s -c \$namespace/config/ora2pg.conf" + ora2pg -p -t \$etype -o \$ltype.sql -b \$namespace/schema/\$\{ltype\}s -c \$namespace/config/ora2pg.conf + ret=`grep "Nothing found" \$namespace/schema/\$\{ltype\}s/\$ltype.sql 2> /dev/null` + if [ ! -z "\$ret" ]; then + rm \$namespace/schema/\$\{ltype\}s/\$ltype.sql + fi +done + +for etype in \$(echo \$SOURCE_TYPE | tr " " "\\n") +do + ltype=`echo \$etype | tr '[:upper:]' '[:lower:]'` + ltype=`echo \$ltype | sed 's/y\$/ie/'` + echo "Running: ora2pg -t \$etype -o \$ltype.sql -b \$namespace/sources/\$\{ltype\}s -c \$namespace/config/ora2pg.conf" + ora2pg -t \$etype -o \$ltype.sql -b \$namespace/sources/\$\{ltype\}s -c \$namespace/config/ora2pg.conf + ret=`grep "Nothing found" \$namespace/sources/\$\{ltype\}s/\$ltype.sql 2> /dev/null` + if [ ! -z "\$ret" ]; then + rm \$namespace/sources/\$\{ltype\}s/\$ltype.sql + fi +done + +echo +echo +echo "To extract data use the following command:" +echo +echo "ora2pg -t COPY -o data.sql -b \$namespace/data -c \$namespace/config/ora2pg.conf" +echo + +exit 0 +}; + close(OUT); + chmod(0700, "$base_path/export_schema.sh"); + + + # Generate shell script to execute all import + print "Creating script import_all.sh to automate all imports.\n"; + my $exportype = "EXPORT_TYPE=\"TYPE " . join(' ', grep( !/^TYPE$/, @SCHEMA_ARRAY)) . "\"\n"; + unless(open(OUT, "> $base_path/import_all.sh")) { + print "FATAL: Can't write to file $base_path/import_all.sh\n"; + exit 1; + } + while (my $l = ) { + $l =~ s/^EXPORT_TYPE=.*/$exportype/s; + $l =~ s/ORA2PG_VERSION/$VERSION/s; + print OUT $l; + } + close(OUT); + chmod(0700, "$base_path/import_all.sh"); +} + +#### +# Set a generic configuration +#### +sub make_config_generic +{ + my $conf_arr = shift; + + chomp(@$conf_arr); + + my $schema = 'CHANGE_THIS_SCHEMA_NAME'; + $schema = $SCHEMA if ($SCHEMA); + for (my $i = 0; $i <= $#{$conf_arr}; $i++) { + if ($IS_MYSQL) { + $conf_arr->[$i] =~ s/^# Set Oracle database/# Set MySQL database/; + $conf_arr->[$i] =~ s/^(ORACLE_DSN.*dbi):Oracle:(.*);sid=SIDNAME/$1:mysql:$2;database=dbname/; + $conf_arr->[$i] =~ s/CHANGE_THIS_SCHEMA_NAME/CHANGE_THIS_DB_NAME/; + $conf_arr->[$i] =~ s/#REPLACE_ZERO_DATE.*/REPLACE_ZERO_DATE\t-INFINITY/; + } elsif ($ENV{ORACLE_HOME}) { + $conf_arr->[$i] =~ s/^ORACLE_HOME.*/ORACLE_HOME\t$ENV{ORACLE_HOME}/; + } + $conf_arr->[$i] =~ s/^USER_GRANTS.*0/USER_GRANTS\t1/; + $conf_arr->[$i] =~ s/^#SCHEMA.*SCHEMA_NAME/SCHEMA\t$schema/; + $conf_arr->[$i] =~ s/^(BINMODE.*)/#$1/; + $conf_arr->[$i] =~ s/^PLSQL_PGSQL.*1/PLSQL_PGSQL\t0/; + $conf_arr->[$i] =~ s/^FILE_PER_CONSTRAINT.*0/FILE_PER_CONSTRAINT\t1/; + $conf_arr->[$i] =~ s/^FILE_PER_INDEX.*0/FILE_PER_INDEX\t1/; + $conf_arr->[$i] =~ s/^FILE_PER_FKEYS.*0/FILE_PER_FKEYS\t1/; + $conf_arr->[$i] =~ s/^FILE_PER_TABLE.*0/FILE_PER_TABLE\t1/; + $conf_arr->[$i] =~ s/^FILE_PER_FUNCTION.*0/FILE_PER_FUNCTION\t1/; + $conf_arr->[$i] =~ s/^TRUNCATE_TABLE.*0/TRUNCATE_TABLE\t1/; + $conf_arr->[$i] =~ s/^DISABLE_SEQUENCE.*0/DISABLE_SEQUENCE\t1/; + $conf_arr->[$i] =~ s/^DISABLE_TRIGGERS.*0/DISABLE_TRIGGERS\t1/; + $conf_arr->[$i] =~ s/^(CLIENT_ENCODING.*)/#$1/; + $conf_arr->[$i] =~ s/^(NLS_LANG.*)/#$1/; + $conf_arr->[$i] =~ s/^#LONGREADLEN.*1047552/LONGREADLEN\t1047552/; + $conf_arr->[$i] =~ s/^AUTODETECT_SPATIAL_TYPE.*0/AUTODETECT_SPATIAL_TYPE\t1/; + $conf_arr->[$i] =~ s/^NO_LOB_LOCATOR.*/NO_LOB_LOCATOR\t0/; + $conf_arr->[$i] =~ s/^USE_LOB_LOCATOR.*/USE_LOB_LOCATOR\t1/; + $conf_arr->[$i] =~ s/^FTS_INDEX_ONLY.*0/FTS_INDEX_ONLY\t1/; + $conf_arr->[$i] =~ s/^DISABLE_UNLOGGED.*0/DISABLE_UNLOGGED\t1/; + $conf_arr->[$i] =~ s/^EMPTY_LOB_NULL.*0/EMPTY_LOB_NULL\t1/; + if ($DSN) { + $conf_arr->[$i] =~ s/^ORACLE_DSN.*/ORACLE_DSN\t$DSN/; + } + if ($DBUSER) { + $conf_arr->[$i] =~ s/^ORACLE_USER.*/ORACLE_USER\t$DBUSER/; + } + if ($DBPWD) { + $conf_arr->[$i] =~ s/^ORACLE_PWD.*/ORACLE_PWD\t$DBPWD/; + } + } + map { s/$/\n/; } @$conf_arr; +} + +__DATA__ +#!/bin/sh +#------------------------------------------------------------------------------- +# +# Script used to load exported sql files into PostgreSQL in practical manner +# allowing you to chain and automatically import schema and data. +# +# Generated by Ora2Pg, the Oracle database Schema converter, version ORA2PG_VERSION +# +#------------------------------------------------------------------------------- + +EXPORT_TYPE="TYPE,TABLE,PARTITION,VIEW,MVIEW,FUNCTION,PROCEDURE,SEQUENCE,TRIGGER,SYNONYM,DIRECTORY,DBLINK" +AUTORUN=0 +NAMESPACE=. +NO_CONSTRAINTS=0 +IMPORT_INDEXES_AFTER=0 +DEBUG=0 +IMPORT_SCHEMA=0 +IMPORT_DATA=0 +IMPORT_CONSTRAINTS=0 +NO_DBCHECK=0 + + +# Message functions +die() { + echo "ERROR: $1" 1>&2 + exit 1 +} + +usage() { + echo "usage: `basename $0` [options]" + echo "" + echo "Script used to load exported sql files into PostgreSQL in practical manner" + echo "allowing you to chain and automatically import schema and data." + echo "" + echo "options:" + echo " -a import data only" + echo " -b filename SQL script to execute just after table creation to fix database schema" + echo " -d dbname database name for import" + echo " -D enable debug mode, will only show what will be done" + echo " -e encoding database encoding to use at creation (default: UTF8)" + echo " -f force no check of user and database existing and do not try to create them" + echo " -h hostname hostname of the PostgreSQL server (default: unix socket)" + echo " -i only load indexes, constraints and triggers" + echo " -I do not try to load indexes, constraints and triggers" + echo " -j cores number of connection to use to import data or indexes into PostgreSQL" + echo " -n schema comma separated list of schema to create" + echo " -o username owner of the database to create" + echo " -p port listening port of the PostgreSQL server (default: 5432)" + echo " -P cores number of tables to process at same time for data import" + echo " -s import schema only, do not try to import data" + echo " -t export comma separated list of export type to import (same as ora2pg)" + echo " -U username username to connect to PostgreSQL (default: peer username)" + echo " -x import indexes and constraints after data" + echo " -y reply Yes to all questions for automatic import" + echo + echo " -? print help" + echo + exit $1 +} + +# Function to emulate Perl prompt function +confirm () { + + msg=$1 + if [ "$AUTORUN" != "0" ]; then + true + else + if [ -z "$msg" ]; then + msg="Are you sure? [y/N/q]" + fi + # call with a prompt string or use a default + read -r -p "${msg} [y/N/q] " response + case $response in + [yY][eE][sS]|[yY]) + true + ;; + [qQ][uU][iI][tT]|[qQ]) + exit + ;; + *) + false + ;; + esac + fi +} + +# Function used to import constraints and indexes +import_constraints () { + if [ -r "$NAMESPACE/schema/tables/INDEXES_table.sql" ]; then + if confirm "Would you like to import indexes from $NAMESPACE/schema/tables/INDEXES_table.sql?" ; then + if [ -z "$IMPORT_JOBS" ]; then + echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/INDEXES_table.sql" + if [ $DEBUG -eq 0 ]; then + psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/INDEXES_table.sql + if [ $? -ne 0 ]; then + die "can not import indexes." + fi + fi + else + echo "Running: ora2pg -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/INDEXES_table.sql" + if [ $DEBUG -eq 0 ]; then + ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/INDEXES_table.sql + if [ $? -ne 0 ]; then + die "can not import indexes." + fi + fi + fi + fi + fi + + if [ -r "$NAMESPACE/schema/tables/CONSTRAINTS_table.sql" ]; then + if confirm "Would you like to import constraints from $NAMESPACE/schema/tables/CONSTRAINTS_table.sql?" ; then + if [ -z "$IMPORT_JOBS" ]; then + echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/CONSTRAINTS_table.sql" + if [ $DEBUG -eq 0 ]; then + psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/CONSTRAINTS_table.sql + if [ $? -ne 0 ]; then + die "can not import constraints." + fi + fi + else + echo "Running: ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/CONSTRAINTS_table.sql" + if [ $DEBUG -eq 0 ]; then + ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/CONSTRAINTS_table.sql + if [ $? -ne 0 ]; then + die "can not import constraints." + fi + fi + fi + fi + fi + + if [ -r "$NAMESPACE/schema/tables/FKEYS_table.sql" ]; then + if confirm "Would you like to import foreign keys from $NAMESPACE/schema/tables/FKEYS_table.sql?" ; then + if [ -z "$IMPORT_JOBS" ]; then + echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/FKEYS_table.sql" + if [ $DEBUG -eq 0 ]; then + psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/tables/FKEYS_table.sql + if [ $? -ne 0 ]; then + die "can not import foreign keys." + fi + fi + else + echo "Running: ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/FKEYS_table.sql" + if [ $DEBUG -eq 0 ]; then + ora2pg$IMPORT_JOBS -c config/ora2pg.conf -t LOAD -i $NAMESPACE/schema/tables/FKEYS_table.sql + if [ $? -ne 0 ]; then + die "can not import foreign keys." + fi + fi + fi + fi + fi + + if [ $NO_CONSTRAINTS -eq 1 ] && [ -r "$NAMESPACE/schema/triggers/trigger.sql" ]; then + if confirm "Would you like to import TRIGGER from $NAMESPACE/schema/triggers/trigger.sql?" ; then + echo "Running: psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/triggers/trigger.sql" + if [ $DEBUG -eq 0 ]; then + psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/triggers/trigger.sql + if [ $? -ne 0 ]; then + die "an error occurs when importing file $NAMESPACE/schema/triggers/trigger.sql." + fi + fi + fi + fi +} + +# Command line options +while getopts "b:d:e:h:j:l:n:o:p:P:t:U:aDfiIsyx?" opt; do + case "$opt" in + a) IMPORT_DATA=1;; + b) SQL_POST_SCRIPT=$OPTARG;; + d) DB_NAME=$OPTARG;; + D) DEBUG=1;; + e) DB_ENCODING=" -E $OPTARG";; + f) NO_DBCHECK=1;; + h) DB_HOST=" -h $OPTARG";; + i) IMPORT_CONSTRAINTS=1;; + I) NO_CONSTRAINTS=1;; + j) IMPORT_JOBS=" -j $OPTARG";; + n) DB_SCHEMA=$OPTARG;; + o) DB_OWNER=$OPTARG;; + p) DB_PORT=" -p $OPTARG";; + P) PARALLEL_TABLES=" -P $OPTARG";; + s) IMPORT_SCHEMA=1;; + t) EXPORT_TYPE=$OPTARG;; + U) DB_USER=" -U $OPTARG";; + x) IMPORT_INDEXES_AFTER=1;; + y) AUTORUN=1;; + "?") usage 1;; + *) die "Unknown error while processing options";; + esac +done + +# Check if post tables import SQL script is readable +if [ ! -z "$SQL_POST_SCRIPT" ]; then + if [ ! -r "$SQL_POST_SCRIPT" ]; then + die "the SQL script $SQL_POST_SCRIPT is not readable." + fi +fi + +# A database name is mandatory +if [ -z "$DB_NAME" ]; then + die "you must give a PostgreSQL database name (see -d option)." +fi + +# A database owner is mandatory +if [ -z "$DB_OWNER" ]; then + die "you must give a username to be used as owner of database (see -o option)." +fi + +# Check if the project directory is readable +if [ ! -r "$NAMESPACE/schema/tables/table.sql" ]; then + die "project directory '$NAMESPACE' is not valid or is not readable." +fi + +# If constraints and indexes files are present propose to import these object +if [ $IMPORT_CONSTRAINTS -eq 1 ]; then + if confirm "Would you like to load indexes, constraints and triggers?" ; then + import_constraints + fi + exit 0 +fi + +# When a PostgreSQL schema list is provided, create them +if [ $IMPORT_DATA -eq 0 ]; then + is_superuser='f' + if [ $NO_DBCHECK -eq 0 ]; then + # Create owner user + user_exists=`psql -d $DB_NAME$DB_HOST$DB_PORT$DB_USER -Atc "select usename from pg_user where usename='$DB_OWNER';"` + is_superuser=`psql -d $DB_NAME$DB_HOST$DB_PORT$DB_USER -Atc "select usesuper from pg_user where usename='$DB_OWNER';"`; + if [ "a$user_exists" = "a" ]; then + if confirm "Would you like to create the owner of the database $DB_OWNER?" ; then + echo "Running: createuser$DB_HOST$DB_PORT$DB_USER --no-superuser --no-createrole --no-createdb $DB_OWNER" + if [ $DEBUG -eq 0 ]; then + createuser$DB_HOST$DB_PORT$DB_USER --no-superuser --no-createrole --no-createdb $DB_OWNER + if [ $? -ne 0 ]; then + die "can not create user $DB_OWNER." + fi + fi + fi + else + echo "Database owner $DB_OWNER already exists, skipping creation." + fi + + # Create database if required + if [ "a$DB_ENCODING" = "a" ]; then + DB_ENCODING=" -E UTF8" + fi + db_exists=`psql -d $DB_NAME$DB_HOST$DB_PORT$DB_USER -Atc "select datname from pg_database where datname='$DB_NAME';"` + if [ "a$db_exists" = "a" ]; then + if confirm "Would you like to create the database $DB_NAME?" ; then + echo "Running: createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME" + if [ $DEBUG -eq 0 ]; then + createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME + if [ $? -ne 0 ]; then + die "can not create database $DB_NAME." + fi + fi + fi + else + if confirm "Would you like to drop the database $DB_NAME before recreate it?" ; then + echo "Running: dropdb$DB_HOST$DB_PORT$DB_USER $DB_NAME" + if [ $DEBUG -eq 0 ]; then + dropdb$DB_HOST$DB_PORT$DB_USER $DB_NAME + if [ $? -ne 0 ]; then + die "can not drop database $DB_NAME." + fi + fi + echo "Running: createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME" + if [ $DEBUG -eq 0 ]; then + createdb$DB_HOST$DB_PORT$DB_USER$DB_ENCODING --owner $DB_OWNER $DB_NAME + if [ $? -ne 0 ]; then + die "can not create database $DB_NAME." + fi + fi + fi + fi + fi + + # When schema list is provided, create them + if [ "a$DB_SCHEMA" != "a" ]; then + nspace_list='' + for enspace in $(echo $DB_SCHEMA | tr "," "\n") + do + lnspace=`echo $enspace | tr '[:upper:]' '[:lower:]'` + if confirm "Would you like to create schema $lnspace in database $DB_NAME?" ; then + echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -c \"CREATE SCHEMA $lnspace;\"" + if [ $DEBUG -eq 0 ]; then + psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -c "CREATE SCHEMA $lnspace;" + if [ $? -ne 0 ]; then + die "can not create schema $DB_SCHEMA." + fi + fi + nspace_list="$nspace_list$lnspace," + fi + done + # Change search path of the owner + if [ "a$nspace_list" != "a" ]; then + if confirm "Would you like to change search_path of the database owner?" ; then + echo "Running: psql$DB_HOST$DB_PORT$DB_USER -d $DB_NAME -c \"ALTER ROLE $DB_OWNER SET search_path TO ${nspace_list}public;\"" + if [ $DEBUG -eq 0 ]; then + psql$DB_HOST$DB_PORT$DB_USER -d $DB_NAME -c "ALTER ROLE $DB_OWNER SET search_path TO ${nspace_list}public;" + if [ $? -ne 0 ]; then + die "can not change search_path." + fi + fi + fi + fi + fi + + # Then import all files from project directory + for etype in $(echo $EXPORT_TYPE | tr "," "\n") + do + + if [ $NO_CONSTRAINTS -eq 1 ] && [ $etype = "TRIGGER" ]; then + continue + fi + + if [ $etype = "GRANT" ] || [ $etype = "TABLESPACE" ]; then + continue + fi + + ltype=`echo $etype | tr '[:upper:]' '[:lower:]'` + ltype=`echo $ltype | sed 's/y$/ie/'` + if [ -r "$NAMESPACE/schema/${ltype}s/$ltype.sql" ]; then + if confirm "Would you like to import $etype from $NAMESPACE/schema/${ltype}s/$ltype.sql?" ; then + echo "Running: psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/${ltype}s/$ltype.sql" + if [ $DEBUG -eq 0 ]; then + psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/schema/${ltype}s/$ltype.sql + if [ $? -ne 0 ]; then + die "an error occurs when importing file $NAMESPACE/schema/${ltype}s/$ltype.sql." + fi + fi + fi + fi + if [ ! -z "$SQL_POST_SCRIPT" ] && [ $etype = "TABLE" ]; then + if confirm "Would you like to execute SQL script $SQL_POST_SCRIPT?" ; then + echo "Running: psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $SQL_POST_SCRIPT" + if [ $DEBUG -eq 0 ]; then + psql --single-transaction $DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $SQL_POST_SCRIPT + if [ $? -ne 0 ]; then + die "an error occurs when importing file $SQL_POST_SCRIPT." + fi + fi + fi + fi + done + + # If constraints and indexes files are present propose to import these object + if [ $NO_CONSTRAINTS -eq 0 ] && [ $IMPORT_INDEXES_AFTER -eq 0 ]; then + if confirm "Would you like to process indexes and constraints before loading data?" ; then + IMPORT_INDEXES_AFTER=0 + import_constraints + else + IMPORT_INDEXES_AFTER=1 + fi + fi + + # When the database owner is not superuser use postgres instead + q_user='postgres' + if [ "$is_superuser" = "t" ]; then + q_user=$DB_OWNER + fi + + # Import objects that need superuser privilege: GRANT and TABLESPACE + if [ -r "$NAMESPACE/schema/grants/grant.sql" ]; then + if confirm "Would you like to import GRANT from $NAMESPACE/schema/grants/grant.sql?" ; then + echo "Running: psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/grants/grant.sql" + if [ $DEBUG -eq 0 ]; then + psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/grants/grant.sql + if [ $? -ne 0 ]; then + die "an error occurs when importing file $NAMESPACE/schema/grants/grant.sql." + fi + fi + fi + fi + if [ -r "$NAMESPACE/schema/tablespaces/tablespace.sql" ]; then + if confirm "Would you like to import TABLESPACE from $NAMESPACE/schema/tablespaces/tablespace.sql?" ; then + echo "Running: psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/tablespaces/tablespace.sql" + if [ $DEBUG -eq 0 ]; then + psql $DB_HOST$DB_PORT -U $q_user -d $DB_NAME -f $NAMESPACE/schema/tablespaces/tablespace.sql + if [ $? -ne 0 ]; then + die "an error occurs when importing file $NAMESPACE/schema/tablespaces/tablespace.sql." + fi + fi + fi + fi +fi + + +# Check if we must just import schema or proceed to data import too +if [ $IMPORT_SCHEMA -eq 0 ]; then + # set the PostgreSQL datasource + pgdsn_defined=`grep "^PG_DSN" config/ora2pg.conf | sed 's/.*dbi:Pg/dbi:Pg/'` + if [ "a$pgdsn_defined" = "a" ]; then + if [ "a$DB_HOST" != "a" ]; then + pgdsn_defined="dbi:Pg:dbname=$DB_NAME;host=$DB_HOST" + else + #default to unix socket + pgdsn_defined="dbi:Pg:dbname=$DB_NAME;" + fi + if [ "a$DB_PORT" != "a" ]; then + pgdsn_defined="$pgdsn_defined;port=$DB_PORT" + else + pgdsn_defined="$pgdsn_defined;port=5432" + fi + fi + + # remove command line option from the DSN string + pgdsn_defined=`echo "$pgdsn_defined" | sed 's/ -. //g'` + + # If data file is present propose to import data + if [ -r "$NAMESPACE/data/data.sql" ]; then + if confirm "Would you like to import data from $NAMESPACE/data/data.sql?" ; then + echo "Running: psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/data/data.sql" + if [ $DEBUG -eq 0 ]; then + psql$DB_HOST$DB_PORT -U $DB_OWNER -d $DB_NAME -f $NAMESPACE/data/data.sql + if [ $? -ne 0 ]; then + die "an error occurs when importing file $NAMESPACE/data/data.sql." + fi + fi + fi + else + # Import data directly from PostgreSQL + if confirm "Would you like to import data from Oracle database directly into PostgreSQL?" ; then + echo "Running: ora2pg$IMPORT_JOBS$PARALLEL_TABLES -c config/ora2pg.conf -t COPY --pg_dsn \"$pgdsn_defined\" --pg_user $DB_OWNER" + if [ $DEBUG -eq 0 ]; then + ora2pg$IMPORT_JOBS$PARALLEL_TABLES -c config/ora2pg.conf -t COPY --pg_dsn "$pgdsn_defined" --pg_user $DB_OWNER + if [ $? -ne 0 ]; then + die "an error occurs when importing data." + fi + fi + fi + fi + + if [ $NO_CONSTRAINTS -eq 0 ] && [ $IMPORT_DATA -eq 0 ]; then + # Import indexes and constraint after data + if [ $IMPORT_INDEXES_AFTER -eq 1 ]; then + import_constraints + fi + fi +fi + +exit 0 + diff --git a/scripts/ora2pg_scanner b/scripts/ora2pg_scanner new file mode 100644 index 0000000000000000000000000000000000000000..26f35c52e0b6df51e4fda873ccf85df6591de9c5 --- /dev/null +++ b/scripts/ora2pg_scanner @@ -0,0 +1,278 @@ +#!/usr/bin/perl +#------------------------------------------------------------------------------ +# Project : Oracle to Postgresql converter +# Name : ora2pg_scanner +# Author : Gilles Darold, gilles _AT_ darold _DOT_ net +# Copyright: Copyright (c) 2000-2020 : Gilles Darold - All rights reserved - +# Function : Script used to scan a list of DSN and generate reports +# Usage : ora2pg_scanner -l dsn_csv_file -o outdir +#------------------------------------------------------------------------------ +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see < http://www.gnu.org/licenses/ >. +# +#------------------------------------------------------------------------------ +use strict; + +use Getopt::Long qw(:config no_ignore_case bundling); + +my $VERSION = '21.1'; + +my @DB_DNS = (); +my $OUTDIR = ''; +my $DRYRUN = 0; +my $INPUT_FILE = ''; +my $CONF_FILE = ''; +my $HELP = 0; +my $ORA2PG_CMD = 'ora2pg'; +my $BINPATH = ''; +my $SEP = ($^O =~ /MSWin32|dos/i) ? "\\" : "/"; +my $COST_UNIT = 5; + +# Collect command line arguments +GetOptions +( + 'c|config=s' => \$CONF_FILE, + 'b|binpath=s' => \$BINPATH, + 'l|list=s' => \$INPUT_FILE, + 't|test!' => \$DRYRUN, + 'o|outdir=s' => \$OUTDIR, + 'u|unit=s' => \$COST_UNIT, + 'h|help!' => \$HELP, +); + +$OUTDIR = 'output' if (!$OUTDIR); + +if (!$INPUT_FILE || !-e $INPUT_FILE || $HELP) +{ + usage(); +} + +if ($BINPATH) +{ + $BINPATH =~ s/\Q$SEP\E$//; + if (-e "$BINPATH$SEP$ORA2PG_CMD") + { + $ORA2PG_CMD = "$BINPATH$SEP$ORA2PG_CMD"; + } + else + { + die "FATAL: path to ora2pg binary must exists: $BINPATH$SEP$ORA2PG_CMD\n"; + } +} + +if ($^O =~ /MSWin32|dos/i) +{ + $ORA2PG_CMD = "perl $ORA2PG_CMD"; +} + +open(IN, $INPUT_FILE) or die "FATAL: can not read file $INPUT_FILE, $!\n"; +while (my $l = ) +{ + #"type","schema/database","dsn","user","password","audit users" + #"MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","mysecret" + #"ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager","hr;system;scott" + # skip header line + chomp($l); + $l =~ s/\r//; + next if ($l !~ /^["]*(MYSQL|ORACLE)["]*,/i); + $l =~ s/"//gs; + my @data = split(/,/, $l); + if ($#data < 4) + { + die "FATAL: wrong number of field at line: $l\n"; + } + my ($type, $schema, $dsn, $user, $passwd, $audit_user) = split(/,/, $l); + push(@DB_DNS, { ( + 'type' => uc($type), + 'schema' => $schema, + 'dsn' => $dsn, + 'user' => $user, + 'pwd' => $passwd, + 'audit_user' => $audit_user, + 'sid' => '', + 'host' => '' + ) + } + ); +} +close(IN); + +# Create the output directory +if (!$DRYRUN) +{ + if (!-d "$OUTDIR") + { + mkdir "$OUTDIR"; + } + else + { + print "FATAL: output directory already exists, $OUTDIR.\n"; + exit 1; + } +} +else +{ + print "Performing connection test only by retrieving the requested schema or database.\n"; +} + +# Start to generate call to ora2pg +my $header = ' --print_header'; +for (my $i = 0; $i < @DB_DNS; $i++) +{ + $header = '' if ($i > 0); + $ENV{ORA2PG_USER} = $DB_DNS[$i]->{user}; + $ENV{ORA2PG_PASSWD} = $DB_DNS[$i]->{pwd}; + # Used to pass additional information to ora2pg command + my $info = ''; + # Set RDBMS type + $info = ' -m' if ($DB_DNS[$i]->{type} eq 'MYSQL'); + # Add custom configuration file if set + $info .= ' -c ' . $CONF_FILE if ($CONF_FILE); + my $cmd_ora2pg = $ORA2PG_CMD . $info; + + my $audit = ''; + $audit = " --audit_user \"$DB_DNS[$i]->{audit_user}\"" if ($DB_DNS[$i]->{audit_user}); + # Extract SID or db name from the DSN + # dbi:Oracle:host=foobar;sid=ORCL;port=1521 + # dbi:Oracle:DB + # dbi:Oracle://192.168.1.10:1521/XE + # DBI:mysql:database=$db;host=$host + if ($DB_DNS[$i]->{dsn} =~ m/(?:sid|database)=([^;]+)/ || + $DB_DNS[$i]->{dsn} =~ m/dbi:Oracle:([\w]+)$/ || + $DB_DNS[$i]->{dsn} =~ m/dbi:Oracle:\/\/[^\/]+\/([\w]+)/ ) + { + $DB_DNS[$i]->{sid} = $1; + } + elsif (!$DB_DNS[$i]->{schema}) + { + print "WARNING: couldn't determine sid/database name for DSN ". $DB_DNS[$i]->{dsn} .", without explicit schema can not processed this entry. Skiping.\n"; + next; + } + else + { + $DB_DNS[$i]->{sid} = 'schema'; + } + + # Extract host + if ($DB_DNS[$i]->{dsn} =~ m/host=([^;]+)/ || $DB_DNS[$i]->{dsn} =~ m/dbi:Oracle:\/\/([^\/]+)/) + { + $DB_DNS[$i]->{host} = $1; + $DB_DNS[$i]->{host} =~ s/:\d$+//; + $DB_DNS[$i]->{host} .= '_'; + } + + # When no schema or database is set, let Ora2Pg autodetect the list of available schema + if ($DB_DNS[$i]->{schema} eq '') + { + if ($DRYRUN) + { + print "Running: $cmd_ora2pg -t SHOW_SCHEMA -s '$DB_DNS[$i]->{dsn}'\n"; + print `$cmd_ora2pg -t SHOW_SCHEMA -s '$DB_DNS[$i]->{dsn}'`; + print "For each schema returned the following commands will be executed:\n"; + print " $cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s '$DB_DNS[$i]->{dsn}' -n '' >> $OUTDIR${SEP}dbs_scan.csv\n"; + print " $cmd_ora2pg -t SHOW_REPORT --dump_as_html --cost_unit_value $COST_UNIT --estimate_cost$audit -s '$DB_DNS[$i]->{dsn}' -n '' >> \"$OUTDIR${SEP}$DB_DNS[$i]->{host}$DB_DNS[$i]->{sid}_-report.html\"\n"; + } + else + { + my @schema_list = `$cmd_ora2pg -t SHOW_SCHEMA -s '$DB_DNS[$i]->{dsn}'`; + foreach my $line (@schema_list) + { + my ($type, $schemaname) = split(/\s+/, $line); + $DB_DNS[$i]->{schema} = $schemaname; + # Escape some chars for file path use + $schemaname = quotemeta($schemaname); + print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> $OUTDIR${SEP}dbs_scan.csv\n"; + `$cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> $OUTDIR${SEP}dbs_scan.csv`; + $header = ''; + + print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_html --cost_unit_value $COST_UNIT --estimate_cost$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> \"$OUTDIR${SEP}$DB_DNS[$i]->{host}$DB_DNS[$i]->{sid}_$schemaname-report.html\"\n"; + `$cmd_ora2pg -t SHOW_REPORT --dump_as_html --cost_unit_value $COST_UNIT --estimate_cost$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> "$OUTDIR${SEP}$DB_DNS[$i]->{host}$DB_DNS[$i]->{sid}_$schemaname-report.html"`; + } + } + } + else + { + # Escape some chars for file path use + my $schemaname = quotemeta($DB_DNS[$i]->{schema}); + + if ($DRYRUN) + { + print "Running: $cmd_ora2pg -t SHOW_SCHEMA -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}'\n"; + print `$cmd_ora2pg -t SHOW_SCHEMA -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}'` + } + else + { + print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> $OUTDIR${SEP}dbs_scan.csv\n"; + `$cmd_ora2pg -t SHOW_REPORT --dump_as_sheet --cost_unit_value $COST_UNIT --estimate_cost$header$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> $OUTDIR${SEP}dbs_scan.csv`; + print "Running: $cmd_ora2pg -t SHOW_REPORT --dump_as_html --cost_unit_value $COST_UNIT --estimate_cost$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> \"$OUTDIR${SEP}$DB_DNS[$i]->{sid}_$schemaname-report.html\"\n"; + `$cmd_ora2pg -t SHOW_REPORT --dump_as_html --cost_unit_value $COST_UNIT --estimate_cost$audit -s '$DB_DNS[$i]->{dsn}' -n '$DB_DNS[$i]->{schema}' >> "$OUTDIR${SEP}$DB_DNS[$i]->{sid}_$schemaname-report.html"`; + } + } +} + +exit 0; + +sub usage +{ + my $msg = shift; + + print "$msg\n" if ($msg); + + print qq{ +Usage: ora2pg_scanner -l CSVFILE [-o OUTDIR] + + -b | --binpath DIR: full path to directory where the ora2pg binary stays. + Might be useful only on Windows OS. + -c | --config FILE: set custom configuration file to use otherwise ora2pg + will use the default: /etc/ora2pg/ora2pg.conf. + -l | --list FILE : CSV file containing a list of databases to scan with + all required information. The first line of the file + can contain the following header that describes the + format that must be used: + + "type","schema/database","dsn","user","password" + + -o | --outdir DIR : (optional) by default all reports will be dumped to a + directory named 'output', it will be created automatically. + If you want to change the name of this directory, set the name + at second argument. + + -t | --test : just try all connections by retrieving the required schema + or database name. Useful to validate your CSV list file. + -u | --unit MIN : redefine globally the migration cost unit value in minutes. + Default is taken from the ora2pg.conf (default 5 minutes). + + Here is a full example of a CSV databases list file: + + "type","schema/database","dsn","user","password" + "MYSQL","sakila","dbi:mysql:host=192.168.1.10;database=sakila;port=3306","root","secret" + "ORACLE","HR","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" + + The CSV field separator must be a comma. + + Note that if you want to scan all schemas from an Oracle instance you just + have to leave the schema field empty, Ora2Pg will automatically detect all + available schemas and generate a report for each one. Of course you need to + use a connection user with enough privileges to be able to scan all schemas. + For example: + + "ORACLE","","dbi:Oracle:host=192.168.1.10;sid=XE;port=1521","system","manager" + + will generate a report for all schema in the XE instance. Note that in this + case the SCHEMA directive in ora2pg.conf must not be set. + +}; + exit 1; +} +