Commite7cb7ee provided basicinfrastructure for allowing a foreign data wrapper or custom scanprovider to replace a join of one or more tables with a scan.However, this infrastructure failed to take into account the needfor possible EvalPlanQual rechecks, and ExecScanFetch would failan assertion (or just overwrite memory) if such a check was attemptedfor a plan containing a pushed-down join.  To fix, adjust the EPQmachinery to skip some processing steps when scanrelid == 0, makingthose the responsibility of scan's recheck method, which also hasthe responsibility in this case of correctly populating the relevantslot.To allow foreign scans to gain control in the right place to makeuse of this new facility, add a new, optional RecheckForeignScanmethod.  Also, allow a foreign scan to have a child plan, which canbe used to correctly populate the slot (or perhaps for somethingelse, but this is the only use currently envisioned).KaiGai Kohei, reviewed by Robert Haas, Etsuro Fujita, and KyotaroHoriguchi.

To support this, we must reconcile some historical anomalies in thebehavior of -c.  In particular, as a backward-incompatibility, -c nolonger implies --no-psqlrc.Pavel Stehule (code) and Catalin Iacob (documentation).  Review byMichael Paquier and myself.  Proposed behavior per Tom Lane.

For unclear reasons, this function doesn't always read the expected datain some old Perl versions.  Rewriting it to avoid use of ARGV seems tododge the problem, and this version is clearer anyway if you ask me.In passing, also improve error message in adjacent append_to_file function.

Commit344cdff made failure to open the target of --output fatal.For consistency, the --log-file switch should behave similarly.Like the previous commit, back-patch to 9.5 but no further.Daniel Verite

The single linked list of memory contexts could result in O(N^2)performance to free a set of contexts if they were not freed inreverse order of creation.  In many cases the reverse order wasused, but there were some significant exceptions that caused real-world performance problems.  Rather than requiring all callers tocare about the order in which contexts were freed, and hunting downand changing all existing cases where the wrong order was used, weadd one pointer per memory context so that the implementationdetails are not so visible.Jan Wieck

The original parallel sequential scan commit included only very limitedchanges to the EXPLAIN output.  Aggregated totals from all workers weredisplayed, but there was no way to see what each individual worker didor to distinguish the effort made by the workers from the effort made bythe leader.Per a gripe by Thom Brown (and maybe others).  Patch by me, reviewedby Amit Kapila.

Peter Geoghegan

Commit32f15d0 fixed this in configure, but missed the similar checkin the MSVC scripts.Michael Paquier, per report from Victor Wagner

At the end of crash recovery, unlogged relations are reset to the emptystate, using their init fork as the template. The init fork is copied tothe main fork without going through shared buffers. Unfortunately WALreplay so far has not necessarily flushed writes from shared buffers todisk at that point. In normal crash recovery, and before theintroduction of 'fast promotions' infd4ced5 / 9.3, theEND_OF_RECOVERY checkpoint flushes the buffers out in time. But withfast promotions that's not the case anymore.To fix, force WAL writes targeting the init fork to be flushedimmediately (using the new FlushOneBuffer() function). In 9.5+ thatflush can centrally be triggered from the code dealing with restoringfull page writes (XLogReadBufferForRedoExtended), in earlier releasesthat responsibility is in the hands of XLOG_HEAP_NEWPAGE's replayfunction.Backpatch to 9.1, even if this currently is only known to trigger in9.3+. Flushing earlier is more robust, and it is advantageous to keepthe branches similar.Typical symptoms of this bug are errors like'ERROR:  index "..." contains unexpected zero page at block 0'shortly after promoting a node.Reported-By: Thom BrownAuthor: Andres Freund and Michael PaquierDiscussion: 20150326175024.GJ451@alap3.anarazel.deBackpatch: 9.1-

ExecOnConflictUpdate() passed t_ctid of the to-be-updated tuple toExecUpdate(). That's problematic primarily because of two reason: Firstand foremost t_ctid could point to a different tuple. Secondly, andthat's what triggered the complaint by Stanislav, t_ctid is changed byheap_update() to point to the new tuple version.  The behavior of AFTERUPDATE triggers was therefore broken, with NEW.* and OLD.* tuplesspuriously identical within AFTER UPDATE triggers.To fix both issues, pass a pointer to t_self of a on-stack HeapTupleinstead.Fixing this bug lead to one change in regression tests, which previouslyfailed due to the first issue mentioned above. There's a reasonableexpectation that test fails, as it updates one row repeatedly within oneINSERT ... ON CONFLICT statement. That is only possible if the secondupdate is triggered via ON CONFLICT ... SET, ON CONFLICT ... WHERE, orby a WITH CHECK expression, as those are executed afterExecOnConflictUpdate() does a visibility check. That could easily beprohibited, but given it's allowed for plain UPDATEs and a rare cornercase, it doesn't seem worthwhile.Reported-By: Stanislav GrozevAuthor: Andres Freund and Peter GeogheganDiscussion: CAA78GVqy1+LisN-8DygekD_Ldfy=BJLarSpjGhytOsgkpMavfQ@mail.gmail.comBackpatch: 9.5, where ON CONFLICT was introduced

Etsuro Fujita

Complete "ALTER POLICY" with a policy name, as we do for DROP POLICY.And, complete "ALTER POLICY polname ON" with a table name that has sucha policy, as we do for DROP POLICY, rather than with any table nameat all.Masahiko Sawada

This module needs explicit initialization in order to replay WAL recordsin recovery, but we had broken this recently following changes to makeother (stranger) scenarios work correctly.  To fix, rework theinitialization sequence so that it always takes place before WAL replaycommences for both master and standby.I could have gone for a more localized fix that just added a "startup"call for the master server, but it seemed better to restructure theexisting callers as well so that the whole thing made more sense.  As adrawback, there is more control logic in xlog.c now than previously, butdoing otherwise meant passing down the ControlFile flag, which seemeduglier as a whole.This also meant adding a check to not re-execute ActivateCommitTs if ithad already been called.Reported by Fujii Masao.Backpatch to 9.5.

More fuzz testing by Andreas Seltenreich exposed that the planner did notcope well with chains of lateral references.  If relation X references Ylaterally, and Y references Z laterally, then we will have to scan X on theinside of a nestloop with Z, so for all intents and purposes X is laterallydependent on Z too.  The planner did not understand this and would generateintermediate joins that could not be used.  While that was usually harmlessexcept for wasting some planning cycles, under the right circumstances itwould lead to "failed to build any N-way joins" or "could not devise aquery plan" planner failures.To fix that, convert the existing per-relation lateral_relids andlateral_referencers relid sets into their transitive closures; that is,they now show all relations on which a rel is directly or indirectlylaterally dependent.  This not only fixes the chained-reference problembut allows some of the relevant tests to be made substantially simplerand faster, since they can be reduced to simple bitmap manipulationsinstead of searches of the LateralJoinInfo list.Also, when a PlaceHolderVar that is due to be evaluated at a join containslateral references, we should treat those references as indirect lateraldependencies of each of the join's base relations.  This prevents us fromtrying to join any individual base relations to the lateral referencesource before the join is formed, which again cannot work.Andreas' testing also exposed another oversight in the "dangerousPlaceHolderVar" test added in commit85e5e22.  Simply rejectingunsafe join paths in joinpath.c is insufficient, because in some caseswe will end up rejecting *all* possible paths for a particular join, againleading to "could not devise a query plan" failures.  The restriction hasto be known also to join_is_legal and its cohort functions, so that theywill not select a join for which that will happen.  I chose to move thesupporting logic into joinrels.c where the latter functions are.Back-patch to 9.3 where LATERAL support was introduced.

ALTER POLICY hadn't fully considered partial policy alternation(eg: change just the roles on the policy, or just change one ofthe expressions) when rebuilding the dependencies.  Instead, itwould happily remove all dependencies which existed for thepolicy and then only recreate the dependencies for the objectsreferred to in the specific ALTER POLICY command.Correct that by extracting and building the dependencies for allobjects referenced by the policy, regardless of if they wereprovided as part of the ALTER POLICY command or were already inplace as part of the pre-existing policy.

I originally modeled this data structure on SpecialJoinInfo, but aftercommitacfcd45 that looks like a pretty poor decision.All we really need is relid sets identifying laterally-referenced rels;and most of the time, what we want to know about includes indirect lateralreferences, a case the LateralJoinInfo data was unsuited to compute withany efficiency.  The previous commit redefined RelOptInfo.lateral_relidsas the transitive closure of lateral references, so that it easily supportschecking indirect references.  For the places where we really do want justdirect references, add a new RelOptInfo field direct_lateral_relids, whichis easily set up as a copy of lateral_relids before we perform thetransitive closure calculation.  Then we can just drop lateral_info_listand LateralJoinInfo and the supporting code.  This makes the planner'shandling of lateral references noticeably more efficient, and shorter too.Such a change can't be back-patched into stable branches for fear ofbreaking extensions that might be looking at the planner's data structures;but it seems not too late to push it into 9.5, so I've done so.

DROP OWNED BY handled GRANT-based ACLs but was not removing roles frompolicies.  Fix that by having DROP OWNED BY remove the role specifiedfrom the list of roles the policy (or policies) apply to, or the entirepolicy (or policies) if it only applied to the role specified.As with ACLs, the DROP OWNED BY caller must have permission to modifythe policy or a WARNING is thrown and no change is made to the policy.

This allows sane behavior in a PGXS build done on a machine where buildtools such as bison are missing.Jim Nasby

As reported in bug #13809 by Alexander Ashurkov, the code for REASSIGNOWNED hadn't gotten word about user mappings.  Deal with them in thesame way default ACLs do, which is to ignore them altogether; they arehandled just fine by DROP OWNED.  The other foreign object cases arealready handled correctly by both commands.Also add a REASSIGN OWNED statement to foreign_data test to exercise theforeign data objects.  (The changes are just before the "cleanup" phase,so it shouldn't remove any existing live test.)Reported by Alexander Ashurkov, then independently by Jaime Casanova.

…melineThis previously resulted in an error and a nonzero exit status, butafter discussion this should rather be a noop with a zero exit status.

Recent releases of libxml2 do not provide error context reports for errorsdetected at the very end of the input string.  This appears to be a bug, orat least an infelicity, introduced by the fix for libxml2'sCVE-2015-7499.We can hope that this behavioral change will get undone before too long;but the security patch is likely to spread a lot faster/further than anyfollow-on cleanup, which means this behavior is likely to be present in thewild for some time to come.  As a stopgap, add a variant regression testexpected-file that matches what you get with a libxml2 that acts this way.

Changing the tablespace of an unlogged relation did not WAL log thecreation and content of the init fork. Thus, after a standby ispromoted, unlogged relation cannot be accessed anymore, with errorslike:ERROR:  58P01: could not open file "pg_tblspc/...": No such file or directoryAdditionally the init fork was not synced to disk, independent of theconfigured wal_level, a relatively small durability risk.Investigation of that problem also brought to light that, even forpermanent relations, the creation of !main forks was not WAL logged,i.e. no XLOG_SMGR_CREATE record were emitted. That mostly turns out notto be a problem, because these files were created when the actualrelation data is copied; nonexistent files are not treated as an errorcondition during replay. But that doesn't work for empty files, andgenerally feels a bit haphazard. Luckily, outside init and main forks,empty forks don't occur often or are not a problem.Add the required WAL logging and syncing to disk.Reported-By: Michael PaquierAuthor: Michael Paquier and Andres FreundDiscussion: 20151210163230.GA11331@alap3.anarazel.deBackpatch: 9.1, where unlogged relations were introduced

Paul Ramsey

These would leak random xlog positions if a walsender used for backup woulda walsender slot previously used by a replication walsender.In passing also fix a couple of cases where the xlog pointer is directlycompared to zero instead of using XLogRecPtrIsInvalid, noted byMichael Paquier.

Previously the "sent" field would be set to 0 and all other xlogpointers be set to NULL if there were no valid values (such as whenin a backup sending walsender).

Commitd5563d7 drew complaints from Coverity, which quitecorrectly complained that one copy of each -c or -f string was beingleaked.  What's more, simple_action_list_append was allocating enough spacefor still a third copy of each string as part of the SimpleActionListCell,even though that coding method had been superseded by a separate strdupoperation.  There were some other minor coding infelicities too.  Thedocumentation needed more work as well, eg it forgot to explain that -ccauses psql not to accept any interactive input.

This has worked that way for a long time, maybe always, but you wouldnot have known it from the documentation.  Also back-patch the notesI added to HEAD earlier today about behavior of the "-f -" switch,which likewise have been valid for many releases.

e3f4cfc introduced a LWLockHeldByMe() call, without the correspondingAssert() surrounding it.Spotted by Coverity.Backpatch: 9.1+, like the previous commit

…ing.Previously, postgres_fdw's connection cache was keyed by user OID andserver OID, but this can lead to multiple connections when it's notreally necessary.  In particular, if all relevant users are mapped tothe public user mapping, then their connection options are certainlythe same, so one connection can be used for all of them.While we're cleaning things up here, drop the "server" argument toGetConnection(), which isn't really needed.  This saves a few cyclesbecause callers no longer have to look this up; the function itselfdoes, but only when establishing a new connection, not when reusingan existing one.Ashutosh Bapat, with a few small changes by me.

This fix accidentally got left out of the previous commit.

Previously, the foreign join pushdown infrastructure left the questionof security entirely up to individual FDWs, but it would be easy fora foreign data wrapper to inadvertently open up subtle security holesthat way.  So, make it the core code's job to determine which usermapping OID is relevant, and don't attempt join pushdown unless it'sthe same for all relevant relations.Per a suggestion from Tom Lane.  Shigeru Hanada and Ashutosh Bapat,reviewed by Etsuro Fujita and KaiGai Kohei, with some furtherchanges by me.

The upcoming patch to allow join pushdown in postgres_fdw needs to usethis code multiple times, which requires moving it to deparse.c.  Thatseems like a good idea anyway, so do that now both on general principleand to simplify the future patch.Inspired by a patch by Shigeru Hanada and Ashutosh Bapat, but I didit a little differently than what that patch did.

Patch-by: Oleksandr ShulginReviewed-by: Craig Ringer and Fujii MasaoBackpatch-through: 9.4 where logical decoding was introduced

listForeignTables' invocation of processSQLNamePattern did not match upwith the other ones that handle potentially-schema-qualified names; itfailed to make use of pg_table_is_visible() and also passed the namearguments in the wrong order.  Bug seems to have been aboriginal in commit0d692a0.  It accidentally sort of worked as long as you didn'tinquire too closely into the behavior, although the silliness was laterexposed by inconsistencies in the test queries added by59efda3(which I probably should have questioned at the time, but didn't).Per bug #13899 from Reece Hart.  Patch by Reece Hart and Tom Lane.Back-patch to all affected branches.

This doesn't add any functionality but just shuffles things around sothat it can be reused and improved later.Author: Fabien CoelhoReviewed-by: Michael Paquier, Álvaro Herrera

Previously, each PGPROC's backendLock was part of the main tranche,and the PGPROC just contained a pointer.  Now, the actual LWLock ispart of the PGPROC.As with previous, similar patches, this makes it significantly easierto identify these lwlocks in LWLOCK_STATS or Trace_lwlocks outputand improves modularity.Author: Ildus KurbangalievReviewed-by: Amit Kapila, Robert Haas

This is following in a long train of similar changes and for the samereasons - seeb319356 andfe702a7 inter alia.Author: Amit KapilaReviewed-by: Alexander Korotkov, Robert Haas

The code that generates a complete SQL query for a given foreign relationwas repeated in two places, and they didn't quite agree: the EXPLAIN caseleft out the locking clause.  Centralize the code so we get the samebehavior everywhere, and adjust calling conventions and which functionsare static vs. extern accordingly .  Centralize the code so we get the samebehavior everywhere, and adjust calling conventions and which functionsare static vs. extern accordingly.Ashutosh Bapat, reviewed and slightly adjusted by me.

Error reported by Igal Sapir.

Author: Michael Paquier

…es, one ofwhich is a preprocessor directive. This leads ecpg to incorrectly parse the comment as nested.

Add- ALTER SYSTEM SET/RESET ... -> GUC variables- ALTER TABLE ... SET WITH -> OIDS- ALTER DATABASE/FUNCTION/ROLE/USER ... SET/RESET -> GUC variables- ALTER DATABASE/FUNCTION/ROLE/USER ... SET ... -> FROM CURRENT/TO- ALTER DATABASE/FUNCTION/ROLE/USER ... SET ... TO/= -> possible valuesAuthor: Fujii MasaoReviewed-by: Michael Paquier, Masahiko Sawada

Dividing INT_MIN by -1 or taking INT_MIN modulo -1 can sometimescause floating-point exceptions or otherwise misbehave.Fabien Coelho and Michael Paquier

Provide per-script statistical info (count of transactions executedunder that script, average latency for the whole script) after amulti-script run, adding an intermediate level of detail to existingglobal stats and per-command stats.Author: Fabien CoelhoReviewer: Michaël Paquier, Álvaro Herrera

This makes the values more stable, which seems like a good thing foranybody who needs to look at at them.Alexander Korotkov and Amit Kapila

KNN GiST with recheck flag should return to executor the same type as orderingoperator, GiST detects this type by looking to return type of function whichimplements ordering operator. But occasionally detecting code works afterreplacing ordering operator function to distance support function.Distance support function always returns float8, so, detecting code get float8instead of actual return type of ordering operator.Built-in opclasses don't have ordering operator which doesn't returnnon-float8 value, so, tests are impossible here, at least now.Backpatch to 9.5 where lozzy KNN was introduced.Author: Alexander KorotkovReport by: Artur Zakirov

…et().All the other jsonb function descriptions refer to the arguments as being"jsonb", but these two said "json".  Make it consistent.  Per bug #13905from Petru Florin Mihancea.No catversion bump --- we can't force one in the back branches, and thisisn't very critical anyway.

Apparently at least one committer hasn't gotten the word that these do notneed to be maintained by hand, since initdb will create them automatically.Noted while fixing bug #13905.No catversion bump since the post-initdb state is exactly the same eitherway.  I don't see a need for back-patch, either.

create_foreignscan_plan needs to know whether any system columns arerequested from a relation (this flag is needed by ForeignNext duringexecution).  However, for join relations this is a pointless test,because it's not possible to request system columns from them, soremove the check.Author: Etsuro FujitaDiscussion:http://www.postgresql.org/message-id/56AA0FC5.9000207@lab.ntt.co.jpReviewed-by: David Rowley, Robert Haas

This field was included in the original definition of the printQueryOptstruct in commita45195a, but it was not used anywhere in thatcommit, nor since then.  Spotted by Dickson S. Guedes.

…ot existPreviously, the first error seen would be that postgresql.conf does notexist.  But for the case where the whole directory does not exist, givean error message about that, together with a hint for how to create one.

Insert sd_notify() calls at server start and stop for integration withsystemd.  This allows the use of systemd service units of type "notify",which greatly simplifies the systemd configuration.Reviewed-by: Pavel Stěhule <pavel.stehule@gmail.com>

Commite09996f was one brick shy of a load: it didn't insistthat the detected JSON number be the whole of the supplied string.This allowed inputs such as "2016-01-01" to be misdetected as valid JSONnumbers.  Per bug #13906 from Dmitry Ryabov.In passing, be more wary of zero-length input (I'm not sure this canhappen given current callers, but better safe than sorry), and do someminor cosmetic cleanup.