module Sequel::Model::Associations::DatasetMethods
Eager loading makes it so that you can load all associated records for a set of objects in a single query, instead of a separate query for each object.
Two separate implementations are provided. eager
should be used most of the time, as it loads associated records using one query per association. However, it does not allow you the ability to filter or order based on columns in associated tables. eager_graph
loads all records in a single query using JOINs, allowing you to filter or order based on columns in associated tables. However, eager_graph
is usually slower than eager
, especially if multiple one_to_many or many_to_many associations are joined.
You can cascade the eager loading (loading associations on associated objects) with no limit to the depth of the cascades. You do this by passing a hash to eager
or eager_graph
with the keys being associations of the current model and values being associations of the model associated with the current model via the key.
The arguments can be symbols or hashes with symbol keys (for cascaded eager loading). Examples:
Album.eager(:artist).all Album.eager_graph(:artist).all Album.eager(:artist, :genre).all Album.eager_graph(:artist, :genre).all Album.eager(:artist).eager(:genre).all Album.eager_graph(:artist).eager(:genre).all Artist.eager(albums: :tracks).all Artist.eager_graph(albums: :tracks).all Artist.eager(albums: {tracks: :genre}).all Artist.eager_graph(albums: {tracks: :genre}).all
You can also pass a callback as a hash value in order to customize the dataset being eager loaded at query time, analogous to the way the :eager_block association option allows you to customize it at association definition time. For example, if you wanted artists with their albums since 1990:
Artist.eager(albums: proc{|ds| ds.where{year > 1990}})
Or if you needed albums and their artist's name only, using a single query:
Albums.eager_graph(artist: proc{|ds| ds.select(:name)})
To cascade eager loading while using a callback, you substitute the cascaded associations with a single entry hash that has the proc callback as the key and the cascaded associations as the value. This will load artists with their albums since 1990, and also the tracks on those albums and the genre for those tracks:
Artist.eager(albums: {proc{|ds| ds.where{year > 1990}}=>{tracks: :genre}})
Public Instance Methods
If the dataset is being eagerly loaded, default to calling all instead of each.
# File lib/sequel/model/associations.rb 2829 def as_hash(key_column=nil, value_column=nil, opts=OPTS) 2830 if (@opts[:eager_graph] || @opts[:eager]) && !opts.has_key?(:all) 2831 opts = Hash[opts] 2832 opts[:all] = true 2833 end 2834 super 2835 end
Adds one or more INNER JOINs to the existing dataset using the keys and conditions specified by the given association. The following methods also exist for specifying a different type of JOIN:
- association_full_join
-
FULL JOIN
- association_inner_join
-
INNER JOIN
- association_left_join
-
LEFT JOIN
- association_right_join
-
RIGHT JOIN
# File lib/sequel/model/associations.rb 2671 def association_join(*associations) 2672 association_inner_join(*associations) 2673 end
If the expression is in the form x = y
where y
is a Sequel::Model
instance, array of Sequel::Model
instances, or a Sequel::Model
dataset, assume x
is an association symbol and look up the association reflection via the dataset's model. From there, return the appropriate SQL
based on the type of association and the values of the foreign/primary keys of y
. For most association types, this is a simple transformation, but for many_to_many
associations this creates a subquery to the join table.
# File lib/sequel/model/associations.rb 2682 def complex_expression_sql_append(sql, op, args) 2683 r = args[1] 2684 if (((op == :'=' || op == :'!=') and r.is_a?(Sequel::Model)) || 2685 (multiple = ((op == :IN || op == :'NOT IN') and ((is_ds = r.is_a?(Sequel::Dataset)) or r.all?{|x| x.is_a?(Sequel::Model)})))) 2686 l = args[0] 2687 if ar = model.association_reflections[l] 2688 if multiple 2689 klass = ar.associated_class 2690 if is_ds 2691 if r.respond_to?(:model) 2692 unless r.model <= klass 2693 # A dataset for a different model class, could be a valid regular query 2694 return super 2695 end 2696 else 2697 # Not a model dataset, could be a valid regular query 2698 return super 2699 end 2700 else 2701 unless r.all?{|x| x.is_a?(klass)} 2702 raise Sequel::Error, "invalid association class for one object for association #{l.inspect} used in dataset filter for model #{model.inspect}, expected class #{klass.inspect}" 2703 end 2704 end 2705 elsif !r.is_a?(ar.associated_class) 2706 raise Sequel::Error, "invalid association class #{r.class.inspect} for association #{l.inspect} used in dataset filter for model #{model.inspect}, expected class #{ar.associated_class.inspect}" 2707 end 2708 2709 if exp = association_filter_expression(op, ar, r) 2710 literal_append(sql, exp) 2711 else 2712 raise Sequel::Error, "invalid association type #{ar[:type].inspect} for association #{l.inspect} used in dataset filter for model #{model.inspect}" 2713 end 2714 elsif multiple && (is_ds || r.empty?) 2715 # Not a query designed for this support, could be a valid regular query 2716 super 2717 else 2718 raise Sequel::Error, "invalid association #{l.inspect} used in dataset filter for model #{model.inspect}" 2719 end 2720 else 2721 super 2722 end 2723 end
The preferred eager loading method. Loads all associated records using one query for each association.
The basic idea for how it works is that the dataset is first loaded normally. Then it goes through all associations that have been specified via eager
. It loads each of those associations separately, then associates them back to the original dataset via primary/foreign keys. Due to the necessity of all objects being present, you need to use all
to use eager loading, as it can't work with each
.
This implementation avoids the complexity of extracting an object graph out of a single dataset, by building the object graph out of multiple datasets, one for each association. By using a separate dataset for each association, it avoids problems such as aliasing conflicts and creating cartesian product result sets if multiple one_to_many or many_to_many eager associations are requested.
One limitation of using this method is that you cannot filter the current dataset based on values of columns in an associated table, since the associations are loaded in separate queries. To do that you need to load all associations in the same query, and extract an object graph from the results of that query. If you need to filter based on columns in associated tables, look at eager_graph
or join the tables you need to filter on manually.
Each association's order, if defined, is respected. If the association uses a block or has an :eager_block argument, it is used.
# File lib/sequel/model/associations.rb 2750 def eager(*associations) 2751 opts = @opts[:eager] 2752 association_opts = eager_options_for_associations(associations) 2753 opts = opts ? Hash[opts].merge!(association_opts) : association_opts 2754 clone(:eager=>opts.freeze) 2755 end
The secondary eager loading method. Loads all associations in a single query. This method should only be used if you need to filter or order based on columns in associated tables, or if you have done comparative benchmarking it and determined it is faster.
This method uses Dataset#graph
to create appropriate aliases for columns in all the tables. Then it uses the graph's metadata to build the associations from the single hash, and finally replaces the array of hashes with an array model objects inside all.
Be very careful when using this with multiple one_to_many or many_to_many associations, as you can create large cartesian products. If you must graph multiple one_to_many and many_to_many associations, make sure your filters are narrow if the datasets are large.
Each association's order, if defined, is respected. eager_graph
probably won't work correctly on a limited dataset, unless you are only graphing many_to_one, one_to_one, and one_through_one associations.
Does not use the block defined for the association, since it does a single query for all objects. You can use the :graph_* association options to modify the SQL
query.
Like eager
, you need to call all
on the dataset for the eager loading to work. If you just call each
, it will yield plain hashes, each containing all columns from all the tables.
# File lib/sequel/model/associations.rb 2778 def eager_graph(*associations) 2779 eager_graph_with_options(associations) 2780 end
Run eager_graph
with some options specific to just this call. Unlike eager_graph
, this takes the associations as a single argument instead of multiple arguments.
Options:
- :join_type
-
Override the join type specified in the association
- :limit_strategy
-
Use a strategy for handling limits on associations. Appropriate :limit_strategy values are:
- true
-
Pick the most appropriate based on what the database supports
- :distinct_on
-
Force use of DISTINCT ON stategy (*_one associations only)
- :correlated_subquery
-
Force use of correlated subquery strategy (one_to_* associations only)
- :window_function
-
Force use of window function strategy
- :ruby
-
Don't modify the
SQL
, implement limits/offsets with array slicing
This can also be a hash with association name symbol keys and one of the above values, to use different strategies per association.
The default is the :ruby strategy. Choosing a different strategy can make your code significantly slower in some cases (perhaps even the majority of cases), so you should only use this if you have benchmarked that it is faster for your use cases.
# File lib/sequel/model/associations.rb 2802 def eager_graph_with_options(associations, opts=OPTS) 2803 opts = opts.dup unless opts.frozen? 2804 associations = [associations] unless associations.is_a?(Array) 2805 ds = if eg = @opts[:eager_graph] 2806 eg = eg.dup 2807 [:requirements, :reflections, :reciprocals, :limits].each{|k| eg[k] = eg[k].dup} 2808 eg[:local] = opts 2809 ds = clone(:eager_graph=>eg) 2810 ds.eager_graph_associations(ds, model, ds.opts[:eager_graph][:master], [], *associations) 2811 else 2812 # Each of the following have a symbol key for the table alias, with the following values: 2813 # :reciprocals :: the reciprocal value to use for this association 2814 # :reflections :: AssociationReflection instance related to this association 2815 # :requirements :: array of requirements for this association 2816 # :limits :: Any limit/offset array slicing that need to be handled in ruby land after loading 2817 opts = {:requirements=>{}, :master=>alias_symbol(first_source), :reflections=>{}, :reciprocals=>{}, :limits=>{}, :local=>opts, :cartesian_product_number=>0, :row_proc=>row_proc} 2818 ds = clone(:eager_graph=>opts) 2819 ds = ds.eager_graph_associations(ds, model, ds.opts[:eager_graph][:master], [], *associations).naked 2820 end 2821 2822 ds.opts[:eager_graph].freeze 2823 ds.opts[:eager_graph].each_value{|v| v.freeze if v.is_a?(Hash)} 2824 ds 2825 end
If the dataset is being eagerly loaded, default to calling all instead of each.
# File lib/sequel/model/associations.rb 2839 def to_hash_groups(key_column, value_column=nil, opts=OPTS) 2840 if (@opts[:eager_graph] || @opts[:eager]) && !opts.has_key?(:all) 2841 opts = Hash[opts] 2842 opts[:all] = true 2843 end 2844 super 2845 end
Do not attempt to split the result set into associations, just return results as simple objects. This is useful if you want to use eager_graph
as a shortcut to have all of the joins and aliasing set up, but want to do something else with the dataset.
# File lib/sequel/model/associations.rb 2851 def ungraphed 2852 ds = super.clone(:eager_graph=>nil) 2853 if (eg = @opts[:eager_graph]) && (rp = eg[:row_proc]) 2854 ds = ds.with_row_proc(rp) 2855 end 2856 ds 2857 end
Protected Instance Methods
Call graph on the association with the correct arguments, update the eager_graph
data structure, and recurse into eager_graph_associations
if there are any passed in associations (which would be dependencies of the current association)
Arguments:
- ds
-
Current dataset
- model
-
Current
Model
- ta
-
table_alias used for the parent association
- requirements
-
an array, used as a stack for requirements
- r
-
association reflection for the current association, or an
SQL::AliasedExpression
with the reflection as the expression and the alias base as the aliaz. - *associations
-
any associations dependent on this one
# File lib/sequel/model/associations.rb 2874 def eager_graph_association(ds, model, ta, requirements, r, *associations) 2875 if r.is_a?(SQL::AliasedExpression) 2876 alias_base = r.alias 2877 r = r.expression 2878 else 2879 alias_base = r[:graph_alias_base] 2880 end 2881 assoc_table_alias = ds.unused_table_alias(alias_base) 2882 loader = r[:eager_grapher] 2883 if !associations.empty? 2884 if associations.first.respond_to?(:call) 2885 callback = associations.first 2886 associations = {} 2887 elsif associations.length == 1 && (assocs = associations.first).is_a?(Hash) && assocs.length == 1 && (pr_assoc = assocs.to_a.first) && pr_assoc.first.respond_to?(:call) 2888 callback, assoc = pr_assoc 2889 associations = assoc.is_a?(Array) ? assoc : [assoc] 2890 end 2891 end 2892 local_opts = ds.opts[:eager_graph][:local] 2893 limit_strategy = r.eager_graph_limit_strategy(local_opts[:limit_strategy]) 2894 2895 if r[:conditions] && !Sequel.condition_specifier?(r[:conditions]) && !r[:orig_opts].has_key?(:graph_conditions) && !r[:orig_opts].has_key?(:graph_only_conditions) && !r.has_key?(:graph_block) 2896 raise Error, "Cannot eager_graph association when :conditions specified and not a hash or an array of pairs. Specify :graph_conditions, :graph_only_conditions, or :graph_block for the association. Model: #{r[:model]}, association: #{r[:name]}" 2897 end 2898 2899 ds = loader.call(:self=>ds, :table_alias=>assoc_table_alias, :implicit_qualifier=>(ta == ds.opts[:eager_graph][:master]) ? first_source : qualifier_from_alias_symbol(ta, first_source), :callback=>callback, :join_type=>local_opts[:join_type], :join_only=>local_opts[:join_only], :limit_strategy=>limit_strategy, :from_self_alias=>ds.opts[:eager_graph][:master]) 2900 if r[:order_eager_graph] && (order = r.fetch(:graph_order, r[:order])) 2901 ds = ds.order_append(*qualified_expression(order, assoc_table_alias)) 2902 end 2903 eager_graph = ds.opts[:eager_graph] 2904 eager_graph[:requirements][assoc_table_alias] = requirements.dup 2905 eager_graph[:reflections][assoc_table_alias] = r 2906 if limit_strategy == :ruby 2907 eager_graph[:limits][assoc_table_alias] = r.limit_and_offset 2908 end 2909 eager_graph[:cartesian_product_number] += r[:cartesian_product_number] || 2 2910 ds = ds.eager_graph_associations(ds, r.associated_class, assoc_table_alias, requirements + [assoc_table_alias], *associations) unless associations.empty? 2911 ds 2912 end
Check the associations are valid for the given model. Call eager_graph_association
on each association.
Arguments:
- ds
-
Current dataset
- model
-
Current
Model
- ta
-
table_alias used for the parent association
- requirements
-
an array, used as a stack for requirements
- *associations
-
the associations to add to the graph
# File lib/sequel/model/associations.rb 2923 def eager_graph_associations(ds, model, ta, requirements, *associations) 2924 return ds if associations.empty? 2925 associations.flatten.each do |association| 2926 ds = case association 2927 when Symbol, SQL::AliasedExpression 2928 ds.eager_graph_association(ds, model, ta, requirements, eager_graph_check_association(model, association)) 2929 when Hash 2930 association.each do |assoc, assoc_assocs| 2931 ds = ds.eager_graph_association(ds, model, ta, requirements, eager_graph_check_association(model, assoc), assoc_assocs) 2932 end 2933 ds 2934 else 2935 raise(Sequel::Error, 'Associations must be in the form of a symbol or hash') 2936 end 2937 end 2938 ds 2939 end
Replace the array of plain hashes with an array of model objects will all eager_graphed associations set in the associations cache for each object.
# File lib/sequel/model/associations.rb 2943 def eager_graph_build_associations(hashes) 2944 hashes.replace(EagerGraphLoader.new(self).load(hashes)) 2945 end
Private Instance Methods
Return a new dataset with JOINs of the given type added, using the tables and conditions specified by the associations.
# File lib/sequel/model/associations.rb 2951 def _association_join(type, associations) 2952 clone(:join=>clone(:graph_from_self=>false).eager_graph_with_options(associations, :join_type=>type, :join_only=>true).opts[:join]) 2953 end
If the association has conditions itself, then it requires additional filters be added to the current dataset to ensure that the passed in object would also be included by the association's conditions.
# File lib/sequel/model/associations.rb 2958 def add_association_filter_conditions(ref, obj, expr) 2959 if expr != SQL::Constants::FALSE && ref.filter_by_associations_add_conditions? 2960 Sequel[ref.filter_by_associations_conditions_expression(obj)] 2961 else 2962 expr 2963 end 2964 end
Return an expression for filtering by the given association reflection and associated object.
# File lib/sequel/model/associations.rb 2986 def association_filter_expression(op, ref, obj) 2987 meth = :"#{ref[:type]}_association_filter_expression" 2988 # Allow calling private association specific method to get filter expression 2989 send(meth, op, ref, obj) if respond_to?(meth, true) 2990 end
Handle inversion for association filters by returning an inverted expression, plus also handling cases where the referenced columns are NULL.
# File lib/sequel/model/associations.rb 2994 def association_filter_handle_inversion(op, exp, cols) 2995 if op == :'!=' || op == :'NOT IN' 2996 if exp == SQL::Constants::FALSE 2997 ~exp 2998 else 2999 ~exp | Sequel::SQL::BooleanExpression.from_value_pairs(cols.zip([]), :OR) 3000 end 3001 else 3002 exp 3003 end 3004 end
Return an expression for making sure that the given keys match the value of the given methods for either the single object given or for any of the objects given if obj
is an array.
# File lib/sequel/model/associations.rb 3009 def association_filter_key_expression(keys, meths, obj) 3010 vals = if obj.is_a?(Sequel::Dataset) 3011 {(keys.length == 1 ? keys.first : keys)=>obj.select(*meths).exclude(Sequel::SQL::BooleanExpression.from_value_pairs(meths.zip([]), :OR))} 3012 else 3013 vals = Array(obj).reject{|o| !meths.all?{|m| o.get_column_value(m)}} 3014 return SQL::Constants::FALSE if vals.empty? 3015 if obj.is_a?(Array) 3016 if keys.length == 1 3017 meth = meths.first 3018 {keys.first=>vals.map{|o| o.get_column_value(meth)}} 3019 else 3020 {keys=>vals.map{|o| meths.map{|m| o.get_column_value(m)}}} 3021 end 3022 else 3023 keys.zip(meths.map{|k| obj.get_column_value(k)}) 3024 end 3025 end 3026 SQL::BooleanExpression.from_value_pairs(vals) 3027 end
Make sure the association is valid for this model, and return the related AssociationReflection
.
# File lib/sequel/model/associations.rb 3030 def check_association(model, association) 3031 raise(Sequel::UndefinedAssociation, "Invalid association #{association} for #{model.name}") unless reflection = model.association_reflection(association) 3032 raise(Sequel::Error, "Eager loading is not allowed for #{model.name} association #{association}") if reflection[:allow_eager] == false 3033 reflection 3034 end
Allow associations that are eagerly graphed to be specified as an SQL::AliasedExpression
, for per-call determining of the alias base.
# File lib/sequel/model/associations.rb 3038 def eager_graph_check_association(model, association) 3039 if association.is_a?(SQL::AliasedExpression) 3040 SQL::AliasedExpression.new(check_association(model, association.expression), association.alias) 3041 else 3042 check_association(model, association) 3043 end 3044 end
Eagerly load all specified associations
# File lib/sequel/model/associations.rb 3047 def eager_load(a, eager_assoc=@opts[:eager]) 3048 return if a.empty? 3049 # Key is foreign/primary key name symbol. 3050 # Value is hash with keys being foreign/primary key values (generally integers) 3051 # and values being an array of current model objects with that specific foreign/primary key 3052 key_hash = {} 3053 # Reflections for all associations to eager load 3054 reflections = eager_assoc.keys.map{|assoc| model.association_reflection(assoc) || (raise Sequel::UndefinedAssociation, "Model: #{self}, Association: #{assoc}")} 3055 3056 # Populate the key_hash entry for each association being eagerly loaded 3057 reflections.each do |r| 3058 if key = r.eager_loader_key 3059 # key_hash for this key has already been populated, 3060 # skip populating again so that duplicate values 3061 # aren't added. 3062 unless id_map = key_hash[key] 3063 id_map = key_hash[key] = Hash.new{|h,k| h[k] = []} 3064 3065 # Supporting both single (Symbol) and composite (Array) keys. 3066 a.each do |rec| 3067 case key 3068 when Array 3069 if (k = key.map{|k2| rec.get_column_value(k2)}) && k.all? 3070 id_map[k] << rec 3071 end 3072 when Symbol 3073 if k = rec.get_column_value(key) 3074 id_map[k] << rec 3075 end 3076 else 3077 raise Error, "unhandled eager_loader_key #{key.inspect} for association #{r[:name]}" 3078 end 3079 end 3080 end 3081 else 3082 id_map = nil 3083 end 3084 3085 loader = r[:eager_loader] 3086 associations = eager_assoc[r[:name]] 3087 if associations.respond_to?(:call) 3088 eager_block = associations 3089 associations = OPTS 3090 elsif associations.is_a?(Hash) && associations.length == 1 && (pr_assoc = associations.to_a.first) && pr_assoc.first.respond_to?(:call) 3091 eager_block, associations = pr_assoc 3092 end 3093 loader.call(:key_hash=>key_hash, :rows=>a, :associations=>associations, :self=>self, :eager_block=>eager_block, :id_map=>id_map) 3094 a.each{|object| object.send(:run_association_callbacks, r, :after_load, object.associations[r[:name]])} if r[:after_load] 3095 end 3096 end
Process the array of associations arguments (Symbols, Arrays, and Hashes), and return a hash of options suitable for cascading.
# File lib/sequel/model/associations.rb 2968 def eager_options_for_associations(associations) 2969 opts = {} 2970 associations.flatten.each do |association| 2971 case association 2972 when Symbol 2973 check_association(model, association) 2974 opts[association] = nil 2975 when Hash 2976 association.keys.each{|assoc| check_association(model, assoc)} 2977 opts.merge!(association) 2978 else 2979 raise(Sequel::Error, 'Associations must be in the form of a symbol or hash') 2980 end 2981 end 2982 opts 2983 end
Return a subquery expression for filering by a many_to_many association
# File lib/sequel/model/associations.rb 3099 def many_to_many_association_filter_expression(op, ref, obj) 3100 lpks, lks, rks = ref.values_at(:left_primary_key_columns, :left_keys, :right_keys) 3101 jt = ref.join_table_alias 3102 lpks = lpks.first if lpks.length == 1 3103 lpks = ref.qualify(model.table_name, lpks) 3104 3105 meths = if obj.is_a?(Sequel::Dataset) 3106 ref.qualify(obj.model.table_name, ref.right_primary_keys) 3107 else 3108 ref.right_primary_key_methods 3109 end 3110 3111 expr = association_filter_key_expression(ref.qualify(jt, rks), meths, obj) 3112 unless expr == SQL::Constants::FALSE 3113 expr = SQL::BooleanExpression.from_value_pairs(lpks=>model.db.from(ref[:join_table]).select(*ref.qualify(jt, lks)).where(expr).exclude(SQL::BooleanExpression.from_value_pairs(ref.qualify(jt, lks).zip([]), :OR))) 3114 expr = add_association_filter_conditions(ref, obj, expr) 3115 end 3116 3117 association_filter_handle_inversion(op, expr, Array(lpks)) 3118 end
Return a simple equality expression for filering by a many_to_one association
# File lib/sequel/model/associations.rb 3122 def many_to_one_association_filter_expression(op, ref, obj) 3123 keys = ref.qualify(model.table_name, ref[:key_columns]) 3124 meths = if obj.is_a?(Sequel::Dataset) 3125 ref.qualify(obj.model.table_name, ref.primary_keys) 3126 else 3127 ref.primary_key_methods 3128 end 3129 3130 expr = association_filter_key_expression(keys, meths, obj) 3131 expr = add_association_filter_conditions(ref, obj, expr) 3132 association_filter_handle_inversion(op, expr, keys) 3133 end
# File lib/sequel/model/associations.rb 3150 def non_sql_option?(key) 3151 super || key == :eager || key == :eager_graph 3152 end
Return a simple equality expression for filering by a one_to_* association
# File lib/sequel/model/associations.rb 3136 def one_to_many_association_filter_expression(op, ref, obj) 3137 keys = ref.qualify(model.table_name, ref[:primary_key_columns]) 3138 meths = if obj.is_a?(Sequel::Dataset) 3139 ref.qualify(obj.model.table_name, ref[:keys]) 3140 else 3141 ref[:key_methods] 3142 end 3143 3144 expr = association_filter_key_expression(keys, meths, obj) 3145 expr = add_association_filter_conditions(ref, obj, expr) 3146 association_filter_handle_inversion(op, expr, keys) 3147 end
Build associations from the graph if eager_graph
was used, and/or load other associations if eager
was used.
# File lib/sequel/model/associations.rb 3156 def post_load(all_records) 3157 eager_graph_build_associations(all_records) if @opts[:eager_graph] 3158 eager_load(all_records) if @opts[:eager] && (row_proc || @opts[:eager_graph]) 3159 super 3160 end