Skip to content

Sort where queries by an association the query filters on - #16853

Open
matrei wants to merge 53 commits into
7.0.xfrom
fix/15589-where-query-association-sort-7.0.x
Open

matrei wants to merge 53 commits into
7.0.xfrom
fix/15589-where-query-association-sort-7.0.x

Conversation

@matrei

@matrei matrei commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #15589.

Sorting a where query by a property of an association failed with a QueryException when the query also had criteria on that association:

Team.where {
    club.name == 'Manchester United'
}.list(sort: 'club.name')
// org.hibernate.QueryException: could not resolve property: club.name of: grails.gorm.tests.Team

The criteria join club under a generated alias, such as club_alias0, and AbstractHibernateQuery records the join under both the association path and the alias. order() found the join, but added the order with the association path, which Hibernate cannot resolve against the root entity. The sort only worked with an explicit alias defined in the where query, because then the sort property already started with the alias. The order is now added with the alias of the join, as club_alias0.name.

The same sort already worked when the query had no criteria on the association, and the Hibernate 7 version of WhereQueryWithAssociationSortSpec on 8.0.x already expects it to work.

A sort by a path through more than one association, such as team.club.name, failed whether or not the where query joined the nested association, because order() split the path at its first dot only. order() now walks the path one association at a time. It reuses a join the criteria registered under the full path, such as team.club, and joins any other association on the path for the sort, registering that join like any other. A sort through an explicit alias, such as t.club.name, works the same way. Because the sort join is registered instead of created anew for every order, a sort by two properties of the same association no longer fails with duplicate association path.

Sorting by an explicit alias produces the same order as before, and so does sorting by the association's id, such as club.id, in a query with criteria on club, which now goes through the join instead of the foreign key.

Sorting no longer leaves out rows

An association that was joined only for a sort was joined with an inner join, so the sort left out the instances whose association is null. Team.list(sort: 'club.name') did not return the teams without a club, Team.findAllWhere([name: 'A team without a club'], [sort: 'club.name']) did not find such a team at all, and the totalCount of a paged list did not count them. A sort should not change which rows a query returns, so the association is now joined with a left join. This applies to where queries, detached criteria and dynamic finders, and to list, find and findAll with an example, findWhere, findAllWhere and a criteria query that is passed pagination arguments.

  • A where query that has criteria on the association still joins it for those criteria, with an inner join by default, and the sort uses that join.
  • A join type that a where query requests for the association, as fetch: [club: JoinType.INNER] does, is used for the sort as well. Before, the sort ignored a requested join type and always created an inner join.
  • Every join of a where query is made when the query is executed, not when the sort, the criteria or the projection that needs it is added. A sort stored on a where query, as in Team.where { ... }.sort('club.name'), is applied before the arguments of list are read and before criteria passed to list are added. A join type that fetch requests there is therefore used for a stored sort as well, and criteria passed to list join the association with their own join type, as they do without the sort. Those criteria can also join it through an alias, as in createAlias('club', 'c'), which failed after a stored sort, and the sort can be by that alias.
  • order in a criteria query can sort by a property of an association too, as in Team.createCriteria().list { order('club.name') }, also by a path through more than one association and inside the criteria of an association. It failed with could not resolve property, and with a Query.Order it sorted by a property of the queried class instead. It joins with a left join, and uses a join that the criteria make for the association. The orders are added once the criteria are built, so an order also uses a join type that the criteria or the fetch argument of list request for its association, and does not keep a collection from being fetched that they ask to fetch, whether the request comes before or after the order. An order by the identifier of an association, such as club.id, makes no join and reads the foreign key, as it did before this PR, so it can still be combined with a projection that groups by the same path or is distinct on it. Inside the criteria of an association, a Query.Order for a single property is resolved against that association too, as an order by a property name is. An order by the identifier of a hasOne, as order('contract.id'), is joined with a left join as well: Hibernate has no foreign key to read it from, and joined it with an inner join, also before this PR.
  • A sort by a path through more than one association keeps the rows too. team.club.id returns a player whose team has no club, as it did before this PR, when only team was joined and club.id was read from its foreign key.
  • A sort does not change what a query fetches either. Hibernate initializes a collection from the rows of a left join that has no condition of its own, so the left join of a sort by a property of a collection, such as players.name, fetched the collection, and a row limit left elements out of it: with max: 2, a team came back with two of its three players. A sort now joins a collection without fetching it, unless the query asks for the collection to be fetched, and the collection is loaded when it is read, as it was before this PR. A collection that the query asks to fetch is fetched with and without the sort. Before this PR, the inner join of the sort took the place of the fetch. When the query fetches another collection, a sort by a collection fetches its collection as well: Hibernate gives a join of a collection that it does not fetch the column aliases of the next collection that it fetches, and fills that one with the elements of the other when both hold elements of the same class. The join of a sort returns a row for each element of the collection, and a bag, a collection declared as a Collection, cannot tell a repeated row from a repeated element, which is why Hibernate does not support fetching more than one bag in a query. So when either collection is a bag, also one fetched through an association or a self-reference, as with fetch: ['publisher.awards': JoinType.LEFT], the query throws an UnsupportedOperationException that names both collections, in list as well. On 7.0.x such a query left out the instances whose sorted collection is empty, and a bag that it fetched could hold its elements more than once. A sort by an association to a single instance is not rejected, and neither is a query that asks to fetch the collection that it sorts by, unless both are bags. The condition that keeps a collection from being fetched is part of the where clause, not of the join condition, which Hibernate combines with the filter of a collection that has a join table, such as the tenant filter of discriminator based multi-tenancy, into SQL that no database accepts.

This changes results on 7.0.x, so section 22 of the upgrade notes describes it, including how to leave the instances out as before. #16863 makes the same change for Hibernate 7 on 8.0.x, where the two versions then return the same rows for these sorts. Apart from this, only queries that failed before behave differently.

A requested join type is applied without a sort or criteria

A where query, a detached criteria, a dynamic finder or a criteria query can request a join type for an association, with fetch: [club: JoinType.INNER] or join('club', JoinType.INNER). In a where query, a detached criteria and a dynamic finder, the type was only used when criteria, or with this PR a sort, joined the association. Otherwise the association was fetched with the outer join of its fetch mode: Team.where { ... }.list(fetch: [club: JoinType.INNER]) returned the teams without a club, and the same query with sort: 'club.name' did not, so a sort still changed the rows of such a query. A requested join type other than LEFT is now applied when the query is executed, whether or not anything else in the query uses the association.

  • JoinType.INNER leaves out the instances whose association is null, from list, get, find, count and the totalCount of a paged list, and still fetches the association.
  • JoinType.RIGHT leaves out the same instances and fetches the association. As a right join does, it also has a row for each associated instance that none of the queried instances refers to: list returns null for such a row and count counts it, unless the criteria of the query exclude it.
  • An inner or a right join requested for a collection keeps the left fetch join, because Hibernate fetches a collection only through a left join, and restricts the query with a subquery that joins the collection. The collection stays fetched, and the count is of instances, not of joined rows. A right join of a collection therefore returns the same instances as an inner join. This also holds when the query is sorted by the collection. The joins of the subquery apply the filters that are enabled for the session, which a check that the collection is not empty does not: with discriminator based multi-tenancy, a shelf whose books all belong to another tenant is left out.
  • A requested path through more than one association, such as fetch: ['team.club': JoinType.INNER], was ignored without criteria on the path. Every association on it is now joined and fetched. A path through a collection, as in fetch: ['players.contract': JoinType.INNER], restricts the query in a subquery, because a join through a collection returns an instance once for each element: count and the totalCount of a paged list count each instance once. The collection on the path is fetched with outer joins and holds all of its elements, also the players without a contract: the join type decides which teams the query returns and never leaves elements out of a fetched collection, which would leave a partial collection in the session. A where query and a dynamic finder return each team once, and a criteria query and a query by example once for each fetched player, as for any fetch of a collection. A JoinType.LEFT for such a path fetches the path too.
  • A query used as a subquery, and a query with a property projection, apply the requested join type too. A subquery applies it once its criteria have been added, as a query that is executed itself does, so an association that the criteria of the subquery join keeps that join.
  • A fetch mode that is requested for the association afterwards replaces the join type: .join('club', JoinType.INNER).list(fetch: [club: 'lazy']) returns the teams without a club, as it did before this PR. The same holds for join('club') or select('club') after join('club', JoinType.INNER), on a where query and in a criteria query, and for the fetch argument of a criteria query. The join that criteria on the association make keeps the requested join type, as it did before this PR, so Team.where { club.name == 'Arsenal' || name == 'Unattached' }.join('club', JoinType.LEFT).list(fetch: [club: 'join']) returns the team without a club.
  • A requested join type is used for the join that the criteria or a projection of a where query make, whether join or the fetch argument of list requests it. The fetch argument is read after the criteria have been applied, so it was not used for such a join: Team.where { club.name == 'Chelsea' || name == 'Unattached First Team' }.list(fetch: [club: JoinType.LEFT]) left out the team without a club.
  • A join type is requested for a nested association by its path. join('team.club', JoinType.LEFT) is used for the join that criteria on team.club make, which was an inner join whatever was requested, and a join type requested for club is no longer used for team.club. Criteria on a nested association can also be used more than once, which failed with duplicate association path.
  • The criteria builder accepted a join type and never used it: join('club', JoinType.INNER) set a fetch mode like the one-argument join, and a JoinType in the fetch argument of list matched no fetch mode, so the association was not fetched at all. Both now join and fetch with the requested type. They are applied once the criteria are built, so an association that the criteria have a block for keeps the join of that block, as does one that the criteria join through an alias, as in createAlias('t.club', 'c').
  • A where query that has criteria on a collection and requests JoinType.LEFT for it uses that join type for the join of the criteria, and no longer fetches the collection through that join, whose rows only have the elements that the criteria match. With join('players', JoinType.LEFT) the collection was initialized with those elements only, and with the fetch argument it would have been since this PR uses the argument for the join of criteria. The collection is now loaded when it is read, with all of its elements. When such a query also fetches another collection, that collection is initialized once the query has run, with a select for each instance, because Hibernate would otherwise read it from the columns of the join of the criteria when both hold elements of the same class. The left join stays, so the query returns the instances without an element when the criteria accept that, as players { kind == null } does.
  • updateAll and deleteAll apply a requested join type too, so they update or delete the instances that list returns. Team.where { ... }.join('club', JoinType.INNER) listed and counted the teams with a club, but deleteAll on it deleted the teams without one as well. An update and a delete cannot join, so each requested join is applied to them as a restriction: a subquery that selects from the entity of the first association on the path and joins the rest of it. MySQL, and MariaDB before version 10.3, do not allow an update or a delete to select from the table that it modifies. On these databases a join that leads back to the queried class, such as the parent of a category, is rejected with an UnsupportedOperationException that says to restrict the query with criteria instead. The check follows the dialect: a MySQL dialect, or a MariaDB dialect older than MariaDB103Dialect. deleteAll selects the identifiers first when the path starts with a collection that has a join table, because Hibernate deletes the join rows before the instances, with the same restriction, which then finds no instance. A one-to-one whose foreign key the other class has is matched through the property that refers back.
  • list, findWhere, findAllWhere, and find and findAll with an example, apply a JoinType in their fetch argument too, before a sort by the association, which uses that join. They ignored it, and since a sort joins with a left join, a sort by the association no longer left out the instances without it either. list fetches the association with the requested join type, returns each instance once when it fetches a collection, and applies RIGHT like INNER, as Hibernate does not support right joins in criteria queries. 'join' and 'eager' in the fetch argument of list now fetch the association with a left join and return every instance, as a where query and a dynamic finder do. They joined it with an inner join without fetching it, so they left out the instances whose association is null or whose collection is empty.
  • A requested join type applies to a collection of a basic type, such as hasMany = [tags: String], too. It was ignored.
  • A right join requested for a collection that criteria on the collection join is made as an inner join, as for a collection that nothing else joins. It returned null for each element that no instance has.
  • The associations before a collection on a requested path, such as team in team.players.contract, are joined and fetched, and so is the collection.
  • A listener of the query events can add criteria on an association to find and findAll with an example, findWhere and findAllWhere. The joins that those criteria need are made after the event; they were left to a query that is not executed, and the query failed.

This also changes results on 7.0.x. Section 23 of the upgrade notes describes it, and the fetching guide now documents the join types.

A criteria query that is passed pagination arguments can now also be sorted by a path through more than one association. Player.createCriteria().list(sort: 'team.club.name', max: 10) { ... } failed when totalCount was read, and with duplicate association path when the criteria used team. The sort now reuses a join that the criteria have for an association path, and creates any other from the root criteria with its whole path, which is how the count recreates it. A join that the criteria make through an alias, as in createAlias('t.club', 'c'), is reused as well. It can also be sorted through an alias that the criteria give an association, as sort: 't.club.name' with createAlias('t.club', 'c'), and by a map of sort properties, which failed with a ClassCastException. An association after the first one on a sort path, or on a requested join path, is joined through the alias of the join before it, so that Hibernate does not read the path through an alias of the criteria that has the name of its first association.

list with a map of sort properties, such as Team.list(sort: ['club.name': 'asc', name: 'desc']), sorted by the last entry only, because each entry replaced the orders of the query. It now sorts by every entry. Hibernate 7 on 8.0.x already does. The same holds for a default sort that the mapping declares for more than one property, of which list applied the last one only. An association is joined once for all the sort properties that use it: list joined it once for each, and find and findAll with an example, findWhere and findAllWhere failed with duplicate association path.

The totalCount of a paged list counts what the list pages through. A list that sorts by a collection returns a row for each element, and max and offset page through those rows, so its total counts them, as on 7.0.x, also when the sort goes through an association that the list fetches. A list that fetches a collection returns each instance once, so its total counts instances, also when a sort joins another collection. A paged query keeps the root alias of a detached criteria when the total is read: new DetachedCriteria(Team, 't').eq('t.name', 'Arsenal').list(max: 2).totalCount failed on 7.0.x.

The "Sorting with Associations" section of the where query docs said that this sort fails and that an explicit alias is required. It now shows sorting by the association path, notes that the path can cross more than one association, that sorting does not change which rows a query returns, and that a sort by an alias that the query does not define fails with IllegalArgumentException: Invalid sort property, as it does on Hibernate 7. It was ignored.

Tests

WhereQueryWithAssociationSortSpec asserted the QueryException. It now checks that a where query with criteria on club can be sorted by club.name:

  • ascending and descending;
  • with a sort map;
  • with chained where calls and fetch: [club: 'join'].

Without the change, these cases fail with the QueryException. It also checks that the order is as before when sorting by club.id, by an explicit alias, and by club.name in a query without criteria on club.

For paths through more than one association, it checks sorting Player by team.club.name with criteria on team.club, with criteria on team only, and without criteria on either, and by t.club.name through an explicit alias of team. It also checks a sort by both club.name and club.id. These cases failed before the change.

For the left join, it checks with a team that has no club that sorts by club.name, club.id, team.club.name and team.club.id keep the team or its player, also through an explicit alias, with criteria on team, and in the totalCount of a paged list. It also checks that a join type requested with fetch is used, and that a criterion club != null leaves the team out. Team.club is nullable in the fixture for that.

For a sort stored on the query with sort, it checks that the team without a club is kept, that a join type requested with fetch in list or find is used, also for team.club.name and in the totalCount of a paged list, and that criteria passed to list return the same rows with and without the sort. Seven of these cases fail without the join being made when the query is executed.

The new AssociationSortSpec checks the same for list, findAll with an example, findWhere, findAllWhere, a criteria query with pagination arguments, and a dynamic finder. It also checks a paged criteria query sorted by team.club.name, with and without criteria on team and with criteria that join team.club through an alias, which failed before, that list applies every property of a default sort, a sort by two properties of one association, and order in a criteria query, including the sequence of several orders, an order that comes before the criteria or the alias of its association, an order by the identifier of an association together with a projection on it, and a Query.Order inside the criteria of an association.

Both specs check a sort by a property of a collection as well: that it keeps a team without players, and that it does not fetch the collection, with a row limit and for a single result, so that no player is left out of it, unless fetch asks for the collection.

56 of the 71 cases of AssociationSortSpec, and 24 of the 59 cases of WhereQueryWithAssociationSortSpec, fail on c0c5a80, the commit before the left join.

The new AssociationJoinTypeSpec checks a join type requested without a sort or criteria on the association: 'join', LEFT, INNER and RIGHT with list, including that the association is fetched, the same rows with and without a sort, count, the totalCount of a paged list, find, get, a dynamic finder, team.club, the players collection, including that an inner and a right join fetch it, also when the query is sorted by it, a path through the collection, which must not leave elements out of it, a fetch mode requested after a join type, a subquery, also one that has criteria on the association it requests a join type for, and a projection. It checks the same for a criteria query, with join and with the fetch argument of list, including count, get, the totalCount of a paged list, a sort, a collection, a nested path and an association the criteria have a block for or join through an alias. It also checks that a join type that fetch requests is used for the join of criteria and of a projection, a join type for a nested association, a fetch mode requested after a join type, and that a path through a collection returns and counts each instance once. For a criteria query that is ordered by an association or a collection, it checks that a join type and a fetch that join or the fetch argument request are applied, before and after the order. It also checks that a left join requested for a collection that a where query has criteria on does not fetch the collection, and that updateAll and deleteAll update or delete the instances that list returns, for an association, a collection, a path through either, and an association whose key the associated class has. 101 of its 135 cases fail without the changes. 28 cases of the two specs fail on 9c59ac4, where an order in a criteria query joined its association before the rest of the criteria was known, and nine fail on e8f7036, where updateAll and deleteAll did not apply a requested join type.

The new MultiTenancyRequestedJoinSpec checks, with discriminator based multi-tenancy, that an inner or a right join requested for a collection leaves out an instance whose elements all belong to another tenant, from a where query, a criteria query and count. Six of its seven cases fail on c5b0edb. DetachedCriteriaJoinTypeSpec in grails-datamapping-core checks that a later join or select replaces a recorded join type and keeps it as one that the fetch mode replaced, also in a copy of the criteria.

The new RequestedJoinBulkOperationDialectSpec checks, with the MySQL and MariaDB dialects, that updateAll and deleteAll reject an inner join requested for an association to the queried class, that the MariaDB 10.3 and H2 dialects apply it, that a join to another class is applied on MySQL, and that criteria on the association restrict the update there. Four of its nine cases fail on cc58a3b.

A second review, with each finding run against 7.0.x, added cases for: deleteAll and updateAll through a collection that has a join table and through the side of a one-to-one that the other class has the foreign key of; a sort by a collection that has a join table and multi-tenant elements, with each kind of query; a sort by a collection while the query fetches another collection of the same class, and the exception when either is a bag; a fetch mode requested after a join type, with criteria on the association; a paged criteria query sorted through an alias, by a map, and by a nested path when an alias has the name of its first association; a join type in the fetch argument of list, findWhere, findAllWhere, and find and findAll with an example, with and without a sort; a collection of a basic type; the associations before a collection on a path; a right join of a collection that criteria join; order('contract.id') on a hasOne; and a projection into an embedded component of an association. The new PreQueryListenerAssociationCriteriaSpec checks criteria on an association that a listener adds. 50 of these cases fail on 1ab83ab. WhereQueryWithAssociationSortSpec also sorts players whose order differs by club name, club id, team name and player name, so that a sort through the wrong join cannot pass.

The new FetchJoinTypeSpec defines the behavior of a requested join type as a chart: for players, players.contract and club with INNER, RIGHT and LEFT, which teams the query returns, what it fetches, that a fetched collection holds all of its elements, whether a team is returned once or once for each fetched player, and the totalCount of a paged list. It checks every row with eight kinds of query: a where query with the fetch argument and with join, a dynamic finder, a criteria query with join and with the fetch argument, findAll with an example, list and a paged list. 28 of its 64 cases fail on 5e6dc8f. AssociationSortSpec has the same kind of chart for a sort by a set, a list or a bag while the query fetches nothing, a set, a list or a bag, with two elements in each collection, since one element cannot show a repeated row.

AssociationJoinTypeSpec has a chart for a left join of a collection that criteria use, with and without another fetched collection, and with criteria that need a value or accept a missing element. It also checks the root alias of a paged detached criteria, including aliases that a subquery could hide. AssociationSortSpec also checks the totalCount of a list sorted by a collection and through an association that it fetches, and the exception for a bag fetched through an association or a self-reference. 20 cases of the two specs fail on 8c4df83.

These pass with the change:

  • all tests of grails-data-hibernate5-core, including the data TCK, and of grails-datamapping-core;
  • codeStyle for grails-data-hibernate5-core and grails-datamapping-core.

Sorting a where query by a property of an association, such as
sort: 'club.name', failed with a QueryException when the query also
had criteria on that association. The criteria join the association
under a generated alias, and the order was added with the association
path, which Hibernate cannot resolve. The order now uses the alias of
the join.

@jdaugherty jdaugherty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally on 0676c67 with :grails-data-hibernate5-core:test. The spec passes, and probes beyond it pass as well: a paged list(max: 2) with totalCount, max plus offset descending, .order('club.name', 'desc') on the detached criteria, a disjunction that uses the association (club.name == 'Arsenal' || name == '...'), a multi-key sort map mixing club.name and name, an explicit alias in the where query combined with a sort by club.name, and ignoreCase: false. The paged count works because HibernateQuery.clone() copies the criterion entries and the subcriteria but not the orders, so the count query carries no order by.

The change is right: getOrCreateAlias registers the join under both the association path and the alias, so resolving the order through criteriaAndAlias.alias covers the generated alias, the explicit alias and the association id in one place, and the detachedCriteria branch closes a latent NPE the old code had for a subquery-backed query. I also confirmed that a sort by an alias the query does not define is silently ignored (the else branch finds no such property), as the docs now say.

One thing to point out: the sort resolves a single association level only. A nested path such as team.club.name still throws, whether or not the where query joins it, while the new docs sentence reads as if any association path works. Details inline on order(). The other inline note is a merge-up heads-up for the Hibernate 7 docs on 8.0.x.

Without an explicit alias, sorting on association properties may fail due to dynamically generated aliases.

For example the following query will fail:
To sort by a property of an associated entity, use the association path. This also works when the `where` query uses the same association:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge-up note: on 8.0.x the Hibernate 7 copy of this guide (grails-data-hibernate7/docs/src/docs/asciidoc/querying/whereQueries.adoc, section "Query Aliases and Sorting") still shows the owner.lastName sort as failing because of a dynamic alias, while the Hibernate 7 WhereQueryWithAssociationSortSpec there already expects it to work. The merge-up carries only this file, so that section stays stale unless it is aligned in the merge-up or a follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. That section was never updated for Hibernate 7. I'll align it in a separate PR against 8.0.x, after checking the Hibernate 7 behavior a bit beyond what its spec covers.

if (createdAssociationPaths.containsKey(sortHead)) {
// sortHead may be either an explicit alias or an association path that
// was joined (with a generated alias) by a where query criterion.
// Hibernate can only resolve the property through the alias.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This resolves one level only: property is split at the first dot, so a sort by a nested path such as team.club.name (Player -> Team -> Club) still throws a QueryException, whether or not the where query joins team.club. It threw before this change too, so it is not a regression, but the new docs sentence ("To sort by a property of an associated entity, use the association path") reads as if any depth works. Either extend this to nested paths, or state the one-level scope in the docs and leave nested paths to a follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it's one level only, as before. I've stated that in the docs now and would leave nested paths for a follow-up. Supporting them means walking the path, reusing the joins that the criteria register under the full path (such as team.club), and adding inner joins for the rest, which I'd rather not do on 7.0.x.

Explicit aliases don't help with nested paths either: the where DSL doesn't recognize def c = team.club, or def t = team; def c = t.club, as an alias, so a query that uses c doesn't compile.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually going to fix this ...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed c0c5a80 to this branch. It reverts b494caa and makes order() walk the sort path one association at a time: a join the criteria registered under the full path, such as team.club, is reused, and each association not joined yet gets an inner join, so team.club.name sorts whether or not the where query joins it, and t.club.name works through an explicit alias of team. A sort by two properties of the same association no longer raises duplicate association path either, since the sort join is registered instead of created for every order. WhereQueryWithAssociationSortSpec covers the nested and duplicate cases (all six failed before the change), and the docs sentence now says the path can cross more than one association.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matrei I think it's worth doing this after looking at it. Take a look at the change and see what you think.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that looks good to me, and it brings Hibernate 5 in line with Hibernate 7, which already sorts by nested paths. I ran the grails-data-hibernate5-core tests on c0c5a80 locally, since CI didn't run them after I removed run ci, and they all pass.

Comparing a few queries before and after the change, the results are the same except for one: with fetch: [owner: JoinType.LEFT] on a nullable association, a sort by owner.name now keeps the rows without an owner, because the sort join uses the requested join type instead of an inner join. I think that's right, but since it changes results on 7.0.x, I'll mention it in the PR description.

@matrei matrei removed the run ci label Nov 4, 2026
matrei and others added 3 commits November 4, 2026 15:07
A sort by a path through more than one association, such as
owner.employer.name, is still not supported.
A sort by a path that crosses more than one association, such as
team.club.name, failed with a QueryException whether or not the where
query joined the nested association, because order() split the path at
its first dot only. The sort property is now qualified with the alias of
the last association on its path: an association the query has already
joined is reused, and one it has not joined yet is joined for the sort
and registered like any other join. A sort by two properties of the same
association no longer fails with "duplicate association path" either,
since the sort join is registered instead of created anew for every
order.
@matrei matrei added the run ci label Nov 4, 2026
@matrei matrei closed this Nov 4, 2026
@matrei matrei reopened this Nov 4, 2026

@matrei matrei left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed c0c5a80 and it looks good to me. Walking the path one association at a time and going through getOrCreateAlias reuses the joins the where query made. It also registers the sort's own join, which fixes the duplicate association path error for two sorts on the same association.

Two minor points, neither blocking:

  • No test covers the LEFT fetch join now being used for the sort.
  • The docs could say that a sort joins an association with an inner join unless fetch asks for another type, so a sort by owner.name leaves out pets without an owner. That was already the case before this change.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorting by an association path reuses a join the where query already made and orders by that join's alias. A path through more than one association works the same way. Not approving. A sort by a nested identifier, such as team.club.id, inner-joins the last association and drops rows whose association is null. Those rows were returned before, because the identifier was ordered through the foreign key.

Since order() walks the whole sort path, a sort by the identifier of an
association that follows another one on the path, such as team.club.id,
joined that last association too. The join is an inner join unless fetch
requests another type, so the sort left out the rows where the
association is null. Before, only the first association was joined and
Hibernate read club.id from the foreign key of team, which kept them.

The identifier is sorted by the foreign key again when the entity before
the association holds it. An association without such a foreign key, a
hasOne or the inverse side of a one-to-one, is still joined, and so is
the first association of a path, as it always has been.

The spec also covers the join type of a sort join: an inner join by
default, and a left join when fetch requests one. Team.club is nullable
in the fixture for these cases. The docs describe the inner join, how to
request a left join, and the identifier of a nested association.

@jdaugherty jdaugherty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed 5e71443 for the feedback on c0c5a80.

@jamesfredley: a sort by a nested identifier keeps its rows again. team.club.id is sorted by the foreign key on team, as it was on 7.0.x, and club is joined only when there is no such foreign key. Details are in the thread.

@matrei: both points are in the same commit.

  • WhereQueryWithAssociationSortSpec now covers the join type of a sort join: an inner join by default, which leaves out a team without a club, and a left join with fetch: [club: JoinType.LEFT], which keeps it. Team.club is nullable in the fixture for that.
  • The docs say that an association is joined with an inner join unless the query requests another type, show the fetch example, and mention the nested identifier.

An association that a where query joined only for a sort was joined with
an inner join, so the sort left out the rows where the association is
null: a sort by club.name or club.id dropped the teams without a club,
from the list and from the total count of a paged list. A sort must not
change which rows a query returns, so the association is now joined with
a left join, unless the query requests another join type for it.

That keeps the rows for every association on a sort path, so the
identifier of a nested association no longer needs to be sorted by its
foreign key, and that special case is removed.

An association that the where query has criteria on is still joined as
the criteria join it, which is an inner join by default.
list, find and findAll with an example, findWhere, findAllWhere and a
criteria query that is passed pagination arguments joined an association
for a sort with an inner join, as where queries did. A sort by
club.name left out the teams without a club, findAllWhere did not find
such a team at all, and the total count of a paged list did not count
it. These sorts now join the association with a left join too, so a sort
no longer changes which instances a query returns.

Dynamic finders sort through the same code as where queries and already
keep those instances. The new spec covers them as well.

The upgrade notes describe the change and how to leave the instances
out as before.

@jdaugherty jdaugherty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to my review above: 2154153 and 1b70845 replace the approach of 5e71443.

A sort should not change which rows a query returns, so an association that is joined only for a sort is now joined with a left join. That keeps the rows for every association on a sort path, so the foreign-key ordering for a nested identifier from 5e71443 is removed again.

@matrei: this goes beyond the two points in your review and changes results on 7.0.x, so I have updated the PR description and added section 22 to the upgrade notes.

  • Where queries, detached criteria and dynamic finders (2154153): without a requested join type the sort join was an inner join, and a sort by club.name or club.id left out the teams without a club, from the list and from the totalCount of a paged list. They are kept now. A join type requested for the association, such as fetch: [club: JoinType.INNER], is used for the sort, and an association the query has criteria on is still joined by the criteria. calculateProjectionPropertyName in the same class already left-joins the associations it joins for a projection, for the same reason.
  • list, find and findAll with an example, findWhere, findAllWhere and a criteria query with pagination arguments (1b70845) had the same inner join in GrailsHibernateQueryUtils and GrailsHibernateUtil. Team.list(sort: 'club.name') left out the teams without a club, and findAllWhere with that sort did not find such a team at all. They use a left join now too.
  • The where query docs say that sorting does not change which rows a query returns. WhereQueryWithAssociationSortSpec and the new AssociationSortSpec cover it, and nine cases of each fail on c0c5a80.

@jamesfredley: the details for team.club.id are in the thread.

A criteria query that is passed pagination arguments and a sort by a
path through more than one association, such as team.club.name, failed
when the total count was read: the sort joined each association from
the criteria of the one before it, and the count recreates every join
from the root criteria by its path, so it looked for club on the root
entity. When the criteria already used the first association of the
path, the query failed with "duplicate association path" instead.

The sort now reuses a join that the criteria have for an association
path, and creates any other from the root criteria with its whole path.
@jdaugherty

Copy link
Copy Markdown
Contributor

@jamesfredley @matrei @borinquenkid, a question that came up while testing this PR and that the PR does not change: what should the fetch argument of the static list() do when the association is null?

With four teams, one of which has no club (Team.club is nullable):

Query 7.0.x (Hibernate 5) 8.0.x (Hibernate 7)
Team.list(fetch: [club: 'join']) 3 teams, club not fetched 3 teams, club fetched
Team.where { ... }.list(fetch: [club: 'join']) 4 teams, fetched 4 teams, fetched
Team.findAllByNameLike('%First%', [fetch: [club: 'join']]) 4 teams, fetched 4 teams, fetched
Team.createCriteria().list(fetch: [club: 'join'], max: 10) { ... } 4 teams, fetched 4 teams, not fetched
Team.findAllWhere([name: 'MU First Team'], [fetch: [club: 'join']]) found, fetched found, not fetched

"Fetched" is Hibernate.isInitialized(team.club) after clearing the session. fetch: [club: 'eager'] behaves like 'join' in list().

So in list() the argument leaves out the team without a club, on both versions:

  • On 7.0.x, GrailsHibernateQueryUtils.populateArgumentsForCriteria turns it into queryRoot.join('club'). That is an inner join and not a fetch, so it only filters.
  • On 8.0.x, HqlListQueryBuilder appends join fetch e.club, an inner fetch join.

It also means that Team.list(sort: 'club.name', fetch: [club: 'join']) still leaves the team out after this PR, because of fetch and not because of the sort.

The options I see:

  1. A left fetch join in list(), which is what the where queries and dynamic finders do. It changes which rows list() returns, like the sort change in this PR, and on 7.0.x list() starts to fetch the association.
  2. A left join without the fetch on 7.0.x, so that the argument no longer filters and still does not fetch.
  3. Leave it as it is and document it.

What do you think should happen, and on which branch? The two rows where Hibernate 7 does not fetch look like a separate gap there.

list with a map of sort properties sorted by the last entry only: each
entry was passed to CriteriaQuery.orderBy, which replaces the orders of
the query. Each order is now added to the ones the query has.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A sort by a nested identifier such as team.club.id now left-joins that association and keeps the rows whose association is null. A paged criteria count can sort by a nested path, and each entry of a sort map is kept. Not approving. A sort stored on the query is applied before the list arguments, so an inner join requested there does not replace the left join the sort already created.

A sort joined an association as soon as the sort was added to the
query. A sort stored on a where query is added before the arguments of
list are read, so the join was already a left join when fetch requested
another type:

    Team.where { ... }.sort('club.name').list(fetch: [club: JoinType.INNER])

returned the teams without a club. Criteria passed to list are added
after a stored sort too, and reused its left join instead of the inner
join they get without the sort.

A sort now only reserves the alias of an association that nothing else
in the query has joined, and the join is made when the query is
executed, with the join type the query requests for the association by
then, or else a left join. Criteria or a projection that need the
association before that join it themselves, under the reserved alias.
A where query, a detached criteria or a dynamic finder can request a
join type for an association, with fetch: [club: JoinType.INNER] or
join('club', JoinType.INNER). The type was only used when criteria or a
sort joined the association. Otherwise the association was fetched with
the outer join of its fetch mode, so the query returned the instances
without the association, and adding a sort to it changed the rows it
returned.

A requested join type other than a left join is now applied when the
query is executed, by joining every association on the requested path
that the query has not joined. Hibernate fetches a collection only
through an outer join, so an inner join requested for a collection
keeps the outer fetch join and restricts the query to the rows whose
collection is not empty.

The fetching guide describes the join types, and section 23 of the
upgrade notes the change in results.
The criteria builder accepted a join type, in join(String, JoinType)
and as a JoinType in the fetch argument of list, and did not use it.
join set a fetch mode, which Hibernate joins with an outer join unless
the association is required, and a JoinType in the fetch argument
matched no fetch mode, so the association was not fetched at all.

Both now join and fetch the association with the requested join type,
as a where query does. The join types requested with join are applied
once the criteria are built, so that an association that the criteria
have a block for keeps the join of that block. An inner join requested
for a collection keeps the outer fetch join and restricts the criteria
to the rows whose collection is not empty.

The fetching guide and section 23 of the upgrade notes now cover the
criteria query.
The subquery that restricts a query to the instances that a requested
join returns compared its own fixed alias with the root alias. With a
detached criteria named join_subquery or join_subquery0 it compared the
subquery with itself, so the restriction matched every row. The alias
is now made from the alias of the root, which it can never equal.
A sort through an association that the list fetches joins from the
fetch. The count drops the fetches, and that join with them, so
Team.list(max: 2, sort: 'club.fans.name', fetch: [club: 'join']) paged
through 7 rows with a total of 3. The fetch and what is joined from it
are now joined again for the count, under aliases of their own: the root
keeps the alias from rendering the list, and Hibernate would generate
the same one for a new join.
The check only looked at the collections of the queried class, so a bag
fetched through an association, as with publisher.awards, or through a
self-reference, as with parent.children, was still read from the
repeated rows of the sort. Both kinds of query now follow what the query
fetches and compare by path.
Making the join of the criteria an inner join lost the instances without
an element when the criteria accept that, as players { kind == null }
does, which 7.0.x returned. The left join stays. The other collections
that the query fetches are now initialized once the query has run, with
a select for each instance, so that Hibernate does not read them from
the columns of the join of the criteria.

This applies when the query requests a left join of a collection that
its criteria use. The upgrade notes and the fetching guide no longer
describe an inner join.

@jdaugherty jdaugherty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed four commits for the review of 8c4df83. The head is a5bfa75.

@jamesfredley:

  • a5bfa75: a requested left join stays a left join, so the squad without players is returned. See the thread on AbstractHibernateQuery.
  • 01ede72: the alias of the subquery is made from the root alias. See the thread on GrailsHibernateUtil.
  • a6346ce, for the count in your summary: a sort join under a fetch is joined again for the count. Team.list(max: 2, sort: 'club.fans.name', fetch: [club: 'join']) paged through 7 rows with a total of 3, and now has a total of 7, the same as without the fetch. A team without a club still counts, and JoinType.INNER leaves it out.
  • bdd9a85, for the bag in your summary: the exception now follows what the query fetches, at any depth, and compares by path. A sort by a collection next to fetch: ['publisher.awards': JoinType.LEFT] or fetch: ['parent.children': JoinType.LEFT] throws when either collection is a bag, in list and in the other kinds of query. The same fetch without a sort by a collection holds each element once.

AssociationSortSpec and AssociationJoinTypeSpec cover the changes, and 20 of their cases fail on 8c4df83. The whole test suite of grails-data-hibernate5-core passes, and so does codeStyle.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A requested left join is no longer turned into an inner join when another collection is fetched. Not approving. A scalar projection of a bag still throws, a mapping fetch below the root still repeats a bag, and a map-valued nested fetch throws.

A query that projects values loads no collection, so a property of a
bag next to a fetched collection threw where 7.0.x returned the values.
The check now leaves a projection alone, in every kind of query.

A fetch that a mapping declares below the root was not looked at, so a
bag that it fetched was read from the repeated rows of a sort. The check
now follows those fetches too, and stops at a mapped association that is
already on the path, so that a class that fetches itself comes to an end.
A fetch path through a map handed the map itself to the reflector, which
threw an IllegalArgumentException.

@jdaugherty jdaugherty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed two commits for the review of a5bfa75. The head is 056588e.

@jamesfredley:

  • 0d97b4c: a query that projects values is no longer rejected. See the thread on GrailsHibernateUtil.
  • 0d97b4c, for the mapping fetch in your summary: the exception now also follows the fetches that a mapping declares below the root, so a sort by a collection next to a bag that fetch: 'join' in a mapping loads throws, as it does for a bag that the query asks to fetch. It stops at a mapped association that is already on the path, so a class that fetches itself comes to an end. list is not affected, because it does not join what a mapping fetches, and fetch: ['publisher.awards': 'select'] turns the mapped fetch off for a query.
  • 056588e, for the map in your summary: a fetch path through a map now goes through its values, so the collections on the path are loaded after the query.

AssociationSortSpec and AssociationJoinTypeSpec cover the changes, and 13 of their cases fail on a5bfa75. The whole test suite of grails-data-hibernate5-core passes, and so does codeStyle.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A scalar projection of a bag no longer throws, and a map-valued nested fetch no longer throws. Not approving. A sort by a collection that the mapping already fetches still throws, and a sort that names the root alias throws.

A sort by a bag that the mapping fetches with fetch: 'join' threw next to
a fetched set or list, because only a fetch that the query requests
counted as a fetch of the sorted collection. The query fetches such a
collection anyway, so the sort makes no join of its own and the query
returns the collections that it returns without the sort. It is now only
rejected next to another bag, as for a requested fetch.

list does not fetch what the mapping fetches, so its sort does join such
a collection without reading it. It accepts the sort as well now, unless
the collection that it fetches is a bag, which the rows of that join
would repeat.
new DetachedCriteria(Team, 't').list(sort: 't.name') threw Invalid sort
property, and 7.0.x ignored the sort. A sort whose first part is the
alias of the root of the query, and not a property of the queried class,
now sorts by the rest of its path, in where queries, detached criteria,
dynamic finders, find and findAll with an example, findWhere and
findAllWhere, and the sort argument of a criteria query.

The sort argument of a criteria query ignored a sort by an alias that the
criteria do not define. It now throws Invalid sort property, as the other
queries do.

@jdaugherty jdaugherty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed two commits for the review of 056588e. The head is 7323d92.

@jamesfredley:

  • 3677b3a: a fetch from the mapping counts as a fetch of the sorted collection. See the thread on GrailsHibernateUtil.
  • 7323d92: a sort can start with the alias of the queried class. See the thread on AbstractHibernateQuery.

28 cases of AssociationSortSpec and UndefinedAliasSortSpec fail on 056588e. The whole test suite of grails-data-hibernate5-core passes, and so does codeStyle.

@testlens-app

testlens-app Bot commented Nov 10, 2026

Copy link
Copy Markdown

✅ All tests passed ✅

🏷️ Commit: 7323d92
▶️ Tests: 57851 executed
⚪️ Checks: 57/57 completed


Learn more about TestLens at testlens.app/docs.

@matrei

matrei commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for all the work on this. The PR has grown from the fix for #15589 into a rework of how GORM for Hibernate 5 joins and fetches associations: 53 commits, 24 files and about 6,200 added lines, and sections 22 and 23 of the upgrade notes list the behavior changes for 7.0.18.

I targeted 7.0.x because the fix was backwards compatible, and most of the PR isn't anymore. Queries return and count other rows than before, updateAll and deleteAll change other rows, some sorts and bulk operations now throw, and grails-datamapping-core changes for every GORM implementation. Hibernate 5 will be around for a while, so getting its joins and fetches right is worth doing, but since Grails 8 has shipped, behavior changes like these have to wait for Grails 9. Each review round has also found new cases in the join and fetch rules, which is another reason to look at them on their own.

I'd like to split it:

If you agree, I'll push the current head to a new branch for the new PR, and then reset this branch to 5e71443.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approving. A sort by an alias is dropped, and a mapped fetch is treated as joined when the fetch depth is 0.

// embedded objects cannot reference entities (at time of writing), so the rest is a component path
break;
}
if (!(persistentProperty instanceof Association)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list(max: 1, sort: ['c.name': 'desc', name: 'asc']) { createAlias('club', 'c'); like('c.name', '%') }, with A in Arsenal and B in Chelsea. The list returns A. The sort by c.name is dropped. Keep that sort.

}
String path = prefix + propertyNames[i];
FetchMode requested = getRequestedFetchMode(root, path);
boolean joined = requested != null ? requested == FetchMode.JOIN :

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hibernate.max_fetch_depth is 0, and group.awards is a bag. Owner.where {}.list(sort: 'children.name') throws. The bag is not joined. Do not treat it as fetched.

@borinquenkid

borinquenkid commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

On 1e475136f4 (the Hibernate 5 half of the list() fetch mode join question): we decided to fix forward instead of reverting #16898 on 8.0.x, tracked in #16939. That keeps Hibernate 5 and Hibernate 7 returning the same rows for Team.list(fetch: [club: 'join']) in 8.0.1 and 7.0.x, so I'd like this commit to stay on 7.0.x.

Two corrections to what I first wrote here: the upgrade note for it is already in this PR (item 23, the bullet on list with 'join' or 'eager'), so nothing needs adding. And I checked taking the commit out: 13 cases of AssociationSortSpec (bag checks, counting each instance once, a sort through a fetched association) fail without it, so it can't be dropped from this PR on its own.

@matrei, when you split this PR: would you keep this commit, with those specs, on 7.0.x? If it goes to 9.0.x with the rest, 8.0.1 ships with Hibernate 5 and Hibernate 7 returning different rows for that call, and #16939 needs another answer.

@jdaugherty

Copy link
Copy Markdown
Contributor

@matrei I agree the PR has become too big. I'd like to propose a different cut: by bug, not by branch. Every part of it fixes a query that returns the wrong rows, modifies the wrong rows, fails, or silently ignores what it was asked, on 7.0.x today. A bug is fixed on the oldest branch that has it and merged up, so each of the PRs below would target 7.0.x.

I targeted 7.0.x because the fix was backwards compatible, and most of the PR isn't anymore.

A fix for a wrong result always changes that result. What we owe applications is an upgrade note that says what changed and how to get the earlier result, and each PR below would carry its own bullets from sections 22 and 23.

since Grails 8 has shipped, behavior changes like these have to wait for Grails 9

That would leave these bugs in every 7.x and 8.x release, including a deleteAll that deletes rows the same query does not list (number 5 below). It would also leave Hibernate 5 and Hibernate 7 returning different rows inside 8.0.x. We decided the same question in #16939 for list(fetch: [club: 'join']): fix forward on the released branch, with an upgrade note and two reviewers, because a revert "would put 8.0.1 back on the less correct behavior".

Each review round has also found new cases in the join and fetch rules, which is another reason to look at them on their own.

Agreed, and that is what the split is for. Nearly all of those findings are in the rules for collections, so those are a PR of their own at the end (number 10), and the rest does not have to wait for it.

The split

Each PR is one bug, with its own specs, docs and upgrade note. They are stacked in this order. "7.0.x" is what the branch does today.

  1. A sort by an association that the query filters on fails (Where queries - Sort by object property #15589). This PR, reset to 5e71443. 7.0.x: Team.where { club.name == 'x' }.list(sort: 'club.name') throws a QueryException, a nested path such as team.club.name throws, and two sort properties of one association fail with duplicate association path.

  2. A sort leaves out rows. 7.0.x: Team.list(sort: 'club.name') does not return the teams without a club, findAllWhere with that sort does not find such a team, and totalCount does not count them. A sort must not change which rows a query returns. This PR covers associations to a single instance, for every kind of query. Keep the rows without an association when sorting by it with Hibernate 7 #16863 is the Hibernate 7 counterpart.

  3. The joins of a where query depend on the order in which the query is put together. 7.0.x: a join type in the fetch argument of list is not used for the join that the criteria make, so Team.where { club.name == 'Chelsea' || name == 'Unattached' }.list(fetch: [club: JoinType.LEFT]) leaves out the team without a club. A join type requested for club is used for team.club. Criteria on a nested association used twice fail with duplicate association path. The fix makes every join when the query is executed, by its full path.

  4. A requested join type is ignored unless something else joins the association. 7.0.x: Team.where { ... }.list(fetch: [club: JoinType.INNER]) returns the teams without a club, and the same query with sort: 'club.name' does not. JoinType.RIGHT is run as an outer fetch, which is not a right join. A nested path, as fetch: ['team.club': JoinType.INNER], is ignored. This holds for where queries, detached criteria, dynamic finders, subqueries and projections. The change in grails-datamapping-core belongs here: join(String) and select(String) replace a join type recorded earlier for the same association.

  5. updateAll and deleteAll ignore a requested join type. 7.0.x: Team.where { ... }.join('club', JoinType.INNER) lists and counts 4 teams, and deleteAll on it deletes 5. Number 4 cannot go in without this one, or it makes the two disagree in more queries. An update or delete cannot join, so the join becomes a subquery, and MySQL does not allow that subquery to select from the table being modified. That one case, a join back to the queried class, throws an UnsupportedOperationException that says what to do, where 7.0.x modified the wrong rows.

  6. The criteria builder accepts a join type and never uses it. 7.0.x: join('club', JoinType.INNER) has had the body of the one-argument join since it was added in 2017, and a JoinType in the fetch argument of list matches no fetch mode, so the association is not fetched at all.

  7. A criteria query cannot be sorted by an association. 7.0.x: order('club.name') throws could not resolve property. A Query.Order inside the criteria of an association sorts by a property of the queried class. A paged criteria query sorted by team.club.name fails when totalCount is read, and one sorted by a map fails with a ClassCastException. The first finding of review 5868034911 (a sort by an alias in such a map is dropped) is fixed here.

  8. list, findWhere, findAllWhere, and find and findAll with an example ignore a join type in fetch. 7.0.x ignores the JoinType. list(fetch: [club: 'join']) joins with an inner join and does not fetch, so it leaves out rows; that is 1e47513, which @borinquenkid asked to keep on 7.0.x for Hibernate 7 list() fetch join (#16898) changed the rows it returns in 8.0.x: add the upgrade note and align Hibernate 5 #16939, and it stays there. The total of a paged detached criteria with a root alias fails.

  9. list with several sort properties sorts by the last one only. 7.0.x: Team.list(sort: ['club.name': 'asc', name: 'desc']) sorts by name, and a default sort that the mapping declares for several properties applies the last. Hibernate 7 sorts by all of them. Independent of the other PRs.

  10. Sorts and joins through a collection. 7.0.x: a sort by a property of a collection leaves out the instances whose collection is empty, and takes the place of a fetch that the query requests for that collection. Criteria on a collection with join('players', JoinType.LEFT) leave the collection in the session with only the elements that matched, 1 of 3. A sort by a collection in a query that fetches a bag returns the bag with its elements repeated, [R1, R1, R2, R2]. The last case has no correct result, for the reason Hibernate does not fetch two bags in one query, so it throws and names both collections. The second finding of review 5868034911 (hibernate.max_fetch_depth 0) and the open thread on bags are settled here.

  11. A sort by an alias that the query does not define is ignored. 7.0.x returns the same order for asc and desc, without an error, also for the alias of the queried class, as in new DetachedCriteria(Team, 't').list(sort: 't.name'). Any other invalid sort property throws IllegalArgumentException: Invalid sort property on 7.0.x already, so an undefined alias now does the same, and the root alias sorts. Hibernate 7 throws a NullPointerException here (Hibernate 7: a sort by an alias that the query does not define throws a NullPointerException #16858), and the same exception is the fix suggested there.

Does this cut work for you? If you see one of these as something other than a bug on 7.0.x, tell me which, and we can look at that one by itself. If it does, I'll reset this PR & split these off.

@borinquenkid

Copy link
Copy Markdown
Member

Does this cut work for you? If you see one of these as something other than a bug on 7.0.x, tell me which, and we can look at that one by itself. If it does, I'll reset this PR & split these off.

@matrei @jdaugherty I'm for the cut by bug, each piece against 7.0.x.

My reasons:

Each piece should carry three things:

  • its own specs
  • its own bullets from sections 22 and 23 of the upgrade notes, saying what changed and how to get the earlier result
  • two human reviewers before it merges

Pieces 5 and 10 add a new UnsupportedOperationException, and 10 also adds a new exception for bags. Those two should name the case and what to do instead, as the current messages do.

For ordering: I have open PRs on the same files (#16890, #16891, #16930, #16931, #17008, #17014). I will rebase them onto whichever of these pieces lands first and won't touch the sort or join(String) code in the meantime.

@matrei

matrei commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@jdaugherty thanks, splitting by bug is better than my cut. A single 9.0.x PR with the other 5,900 lines would have the same problem this one has. One PR per bug, stacked, each with its own specs, docs and note: agreed. Item 1 as this PR, reset to 5e71443: agreed, with one change that I describe below the table.

Where we still differ is the branch, and I don't think "every item is a bug" settles it. The template says patch branches take bug fixes only, but it doesn't say whether a fix that changes the result of a query that runs today counts as one. I'd use this test for each item:

  • 7.0.x when a query that threw now works, or a query keeps its rows and is now sorted the way it asks to be (a sort that was dropped, or applied to another property).
  • 9.0.x when a query that runs today returns, counts, updates or deletes other rows, or now throws.

Why I don't think an upgrade note is enough on 7.0.x:

  • The notes for 7.0.18 are sections 22 and 23 of "Upgrading from Grails 6 to Grails 7". Nobody going from 7.0.17 to 7.0.18 reads that, and patch bumps are often merged by Dependabot or Renovate without a look.
  • Several of the old results aren't arbitrary. JPA defines path navigation with inner join semantics (section 4.4.4 of the spec), which is what a sort by club.name does today, and list(fetch: [club: 'join']) has used an inner join since 2018 (3ac60b2). The new behavior is better, but it's a choice of semantics, and changing one is what a major release is for.
  • The fixes also give queries that run today new failures and new query shapes: an UnsupportedOperationException for two bags or for a bulk join on MySQL, subqueries instead of joins, and a select for each instance to initialize collections after the query. Correct or not, that's more than a patch release should change.
  • "A revert would put 8.0.1 back on the less correct behavior" (Hibernate 7 list() fetch join (#16898) changed the rows it returns in 8.0.x: add the upgrade note and align Hibernate 5 #16939) holds for every behavior change. If it's enough, no change ever has to wait for a major.

Applied to your list:

# Bug Branch
1 Sort by an association the query filters on fails 7.0.x, this PR at 5e71443, with the change below
2 A sort leaves out rows 9.0.x, together with #16863, including a sort that uses a join type the query requests
3 Joins depend on the order the query is put together 9.0.x. On 7.0.x only the duplicate association path for nested criteria used twice, if it can be fixed without changing which join type wins
4 A requested join type is ignored unless something else joins 9.0.x
5 updateAll and deleteAll ignore a requested join type 9.0.x, with 4. A NOTE in the 7.0.x fetching guide that they ignore it, and to restrict with criteria instead, would help in the meantime
6 The criteria builder ignores a join type 9.0.x
7 A criteria query can't be sorted by an association 7.0.x for what threw (order('club.name'), the paged count, the sort map's ClassCastException) and what sorted by another property (Query.Order inside an association), with the join the sort argument of a criteria query uses on 7.0.x today, an inner join, so 7.0.x sorts the same way everywhere. 9.0.x makes them all left joins with item 2
8 list, findWhere and query by example ignore a join type in fetch 9.0.x, including 1e47513 (see below). The totalCount of a paged detached criteria with a root alias, which throws, on 7.0.x
9 list with several sort properties sorts by the last one 7.0.x
10 Sorts and joins through a collection 9.0.x, once we agree what a sort by a collection should do (see below)
11 A sort by an undefined alias is ignored 9.0.x, decided together with #16858. A sort by the root alias (t.name), which is dropped today, on 7.0.x

Item 1 at 5e71443 doesn't fully pass that test. Without criteria on the association, a sort by it works on 7.0.x and joins it with an inner join, even when the query requests a left join. 5e71443 uses the requested join type for the sort, so a query that works today returns more rows:

// Rex (owner Bob), Tom (owner Ann), Stray (no owner)
Pet.where { name ==~ '%' }.list(sort: 'owner.name', fetch: [owner: JoinType.LEFT])*.name
// 7.0.x:      [Tom, Rex]
// 5e71443864: [Stray, Tom, Rex]

The new result is the better one, since the same query without the sort returns Stray, but it's the change of item 2. So after the reset I'll make a join that only the sort needs an inner join again, as on 7.0.x, and take the fetch: [owner: JoinType.LEFT] example out of the where query guide. Item 1 then changes only queries that threw.

Two more things that point the same way:

The change in grails-datamapping-core changes Hibernate 7 on 8.0.x. join(String) and select(String) now remove the join type from getJoinTypes(). Hibernate 7 reads that map in QueryAliasRegistrar.registerDetachedJoins and JpaCriteriaQueryCreator.applyEagerFetchJoins, and doesn't read the new getSupersededJoinTypes(). After the merge-up, join('club', JoinType.INNER).join('club') on Hibernate 7 fetches with a left join instead of the inner join, and the join of criteria on club loses the requested type, which Hibernate 5 keeps through getSupersededJoinTypes(). That's from reading the code, I haven't run it, and this PR has no Hibernate 7 test that would catch it. It's also a public API addition on a patch branch, so it belongs with item 4 on 9.0.x, with Hibernate 7 covered.

1e47513 and #16898. @borinquenkid, I'd rather revert #16898 on 8.0.x before 8.0.1 is cut, keep it on 9.0.x when the revert is merged up (it's already there), and land 1e47513 with item 8 on 9.0.x. Then Hibernate 5 and 7 return the same rows in 8.0.1, as in 8.0.0, and the same rows in 9.0, and #16940 becomes a 9.0.x note. If we keep #16898 in 8.0.1 anyway, 1e47513 should go to 8.0.x on its own rather than to 7.0.x: 7.0.x has no Hibernate 7 to agree with. It's a three-line change, and the 13 cases of AssociationSortSpec that need it were written for the later commits and move with them.

For the 9.0.x part:

  • Hibernate 5 and 7 change together, in the same PR or in a pair, against the same spec. The charts in FetchJoinTypeSpec and AssociationSortSpec are the best description of the rules we have. I'd put them in an issue and agree on them before more code, starting with item 10: whether a sort by a property of a collection, which returns a row for each element, should be supported for an entity query at all.
  • Each PR stays on its bug. This one went through 13 review rounds and grew whenever a fix also took on an older issue nearby (4646683, e6343c4 and 0c81f00, for example), which gave the next round more to find. In the split PRs, a regression of the PR is fixed in it, and an older issue that a review finds gets an issue of its own.

If this works for you, please push 7323d92 to a branch before resetting, so nothing is lost. Since the same branch question came up on #16898 and #16939, I'll also start a thread on dev@ so we settle it once and don't argue it per PR.

@jdaugherty

Copy link
Copy Markdown
Contributor

@matrei thanks, we agree on more than we differ on: one PR per bug, stacked, each with its own specs, docs and note; a regression of a PR is fixed in it, and an older issue that a review finds gets an issue of its own. I also agree with your change to item 1, see below.

On the branch:

9.0.x when a query that runs today returns, counts, updates or deletes other rows, or now throws.

That test protects an application whose query runs today. It does not ask whether the query does what it was asked, and for these items it does not:

  • Team.where { ... }.join('club', JoinType.INNER) lists and counts 4 teams, and deleteAll() on it deletes 5. That query runs today, and it deletes a row that the application did not select. Under the test it stays that way in every 7.x and 8.x release.
  • Team.where { ... }.list(fetch: [club: JoinType.INNER]) returns the teams without a club, and the same query with sort: 'club.name' does not. One of the two results is wrong whichever semantics we prefer.
  • A sort by a collection in a query that fetches a bag returns the bag as [R1, R1, R2, R2].

The policy we have is the one in the template: patch branches take bug fixes. A bug fix to a query that runs always changes what that query returns, so a test that excludes those excludes most fixes to queries. Your own table shows that: items 9 and the root alias of 11 change the result of queries that run today, and you put them on 7.0.x, rightly.

Nobody going from 7.0.17 to 7.0.18 reads that

Agreed, the place is wrong. That is a reason to move the notes, not the fixes: a section for changes within 7.0.x at the top of the upgrade guide, and each PR titled so that the drafted release notes name the change. I'll do that in the split.

JPA defines path navigation with inner join semantics (section 4.4.4 of the spec) ... list(fetch: [club: 'join']) has used an inner join since 2018

Section 4.4.4 is about path expressions in a JPQL query. A sort argument is not one, and GORM does not present it as one: the same query without the sort returns the row. What makes these bugs and not a choice of semantics is that 7.0.x disagrees with itself. 'join' returns every instance from a where query, a dynamic finder and a criteria query, and leaves instances out from list. A requested join type is applied when criteria join the association and ignored when they do not. How long that has been so does not define it: the criteria builder's join(property, JoinType) has ignored its second argument since 2017.

new failures and new query shapes

Each new exception replaces a wrong result, not a right one: repeated elements in a bag, and an update or delete of rows that the query does not return. I take the point on the query shapes, and each PR should say in its description which SQL changes, so a reviewer can weigh it.

"A revert would put 8.0.1 back on the less correct behavior" (#16939) holds for every behavior change.

It holds for a bug. A feature, or a deliberate change of a defined behavior, has no less correct side, and those wait for a major.

Moving the rest to 9.0.x also skips 8.x: Hibernate 5 ships in 8.0.x and 8.1.x, and they would keep all of this until 9.

Where you are right:

  • Item 1. Pet.where { name ==~ '%' }.list(sort: 'owner.name', fetch: [owner: JoinType.LEFT]) returning Stray is item 2's change. Please make a join that only the sort needs an inner join again in this PR and take the example out, so that item 1 only changes queries that threw.
  • grails-datamapping-core and Hibernate 7. Hibernate 7 on 8.0.x reads getJoinTypes() in JoinPlanner, JpaCriteriaQueryCreator and QueryAliasRegistrar, and nothing there reads getSupersededJoinTypes(). I have not run it either. Item 4 does not merge up until Hibernate 7 has the same rule and a spec for it: a fetch mode requested afterwards replaces the join type, and the join of criteria on the association keeps it. The accessor is an addition, and the fix needs it across two modules; if that is the objection, I'd rather mark it as internal than move the fix.
  • The stack releases together. Cutting the branches showed that five spec cases pass on 7.0.x, fail after item 2 and pass again later, all of them a sort together with fetch: [club: JoinType.INNER], where the inner join of the sort hid that the join type was ignored. So items 2 to 8 must be in the same release.
  • Item 10. I'll open an issue with the charts of FetchJoinTypeSpec and AssociationSortSpec, so we agree on what a sort by a collection does before that PR is reviewed. It is last in the stack, and the others do not wait for it.

On 1e47513 and #16898: @borinquenkid and I would keep both where they are, for the reason above.

Where it stands: the branches are cut locally, each on top of the one before it, and the last one has the tree of 7323d92 plus fixes for both findings of review 5868034911 (the alias sort in a paged criteria query in the criteria sort PR, hibernate.max_fetch_depth in the collection PR). The order differs a little from my list: the sort map fix comes third, since it is independent, and updateAll and deleteAll come right after the requested join type. I'll push 7323d92 to a branch of its own before this PR is reset.

A thread on dev@ is the right place to settle the branch question.

@matrei

matrei commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@jdaugherty thanks, that's a lot of common ground, and agreed that dev@ settles the branch question: https://lists.apache.org/thread.html/lsg5glbf0f4xjsppm521c7vw9yrh64r0. So here only what doesn't depend on it.

The deleteAll example. I ran it on 7.0.x, with five teams named ... First Team, four of them with a club:

Query on 7.0.x list count deleteAll
Team.where { name ==~ '%First%' }.join('club', JoinType.INNER) 5 5 5
Team.where { club.name ==~ '%' || name == 'Unattached First Team' } 4 4 throws
The same with .join('club', JoinType.INNER) 4 4 throws

The first query agrees with itself on 7.0.x today, because list and count ignore the join type there too (item 4). It lists 4 teams and deletes 5 only once item 4 applies the join type to list and count, which is what you meant by "Number 4 cannot go in without this one". With criteria on club, deleteAll on 7.0.x throws Joins cannot be used in a DELETE or UPDATE operation instead of deleting another row. So I read item 5 as the other half of item 4 rather than a fix for rows that 7.0.x deletes today, and it might help reviewers if its PR and note put it that way. Tell me if I've missed a case where they disagree on 7.0.x.

On items 9 and 11: they change the order of rows that the caller asked to have sorted, not which rows come back. That's the line in my test, and the one I proposed on dev@.

The order of the stack. Several pieces are on 7.0.x in both of our tables:

  • item 1;
  • item 9;
  • the parts of item 7 that threw or sorted by another property;
  • the totalCount of a paged detached criteria with a root alias, from item 8;
  • the sort by the root alias, from item 11;
  • the duplicate association path of nested criteria, from item 3, if it comes apart from the rest.

Could those go first in the stack, before item 2? Then they can be reviewed and merged now, whatever dev@ decides, and items 2 to 8, which have to be released together anyway, follow as one block. For item 7 that means sorting with the join that the sort argument of a criteria query uses on 7.0.x today, an inner join. Item 2 then makes it a left join along with every other sort, on whichever branch item 2 lands.

Item 4 and the merge-up. A merge-up can't leave a commit behind. If item 4 goes to 7.0.x, the next merge-up into 8.0.x either waits for Hibernate 7 or brings the change to Hibernate 7 without its spec. So the Hibernate 7 rule and its spec need to be on 8.0.x before item 4 merges into 7.0.x.

Agreed:

  • a section for the changes within 7.0.x at the top of the upgrade guide, and PR titles that name the change for the release notes;
  • the SQL changes in each PR description;
  • the issue for item 10 before its PR is reviewed;
  • two reviewers for each piece;
  • pushing 7323d92 to a branch of its own before the reset.

I'll make the item 1 change after the reset.

@jdaugherty

Copy link
Copy Markdown
Contributor

@matrei you're right about deleteAll, and thanks for running it. I get the same on 7.0.x: with join('club', JoinType.INNER) and no criteria on club, list, count, updateAll and deleteAll all return or change 5, and with criteria on club the bulk operations throw Joins cannot be used in a DELETE or UPDATE operation. The 4 against 5 that I described only exists once item 4 applies the join type to list and count. So item 5 is the other half of item 4, and its PR and note will say so. I've corrected it on dev@ as well.

The one place I found where they disagree on 7.0.x is a stored sort: Team.where { name ==~ '%Team' }.sort('club.name') lists 4 of 5 teams, count() on it throws a SQLGrammarException, and updateAll on it changes 5. That is item 2.

On the merge-up: agreed, a merge-up can't leave a commit behind. The Hibernate 7 rule and its spec go to 8.0.x before item 4 merges into 7.0.x.

On the order of the stack: I'll move the sort map (item 9) and the totalCount of a paged detached criteria with a root alias ahead of item 2, since both stand alone. The other pieces you list need a variant that differs from what is cut now (the criteria sort with an inner join, the root alias sort without the undefined alias check), so I'd wait for dev@ before cutting those.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants