How to Update Nested Dictionaries in Python Without Overwriting Existing Keys

Nested dictionaries are a cornerstone of Python programming, enabling the storage of hierarchical and structured data—think configuration files, JSON responses, or multi-level datasets. A common challenge arises when updating these nested dictionaries: how to modify specific keys without overwriting existing nested data. For example, if you’re merging user settings with default configurations, you wouldn’t want to lose existing user preferences buried deep in the hierarchy.

This blog explores practical methods to update nested dictionaries while preserving existing keys, from manual recursive/iterative functions to leveraging third-party libraries. Whether you’re a beginner or an experienced developer, you’ll learn how to handle even the most complex nested structures safely.

Table of Contents#

  1. Understanding Nested Dictionaries
  2. The Pitfall of Shallow Updates
  3. Method 1: Recursive Deep Update
  4. Method 2: Iterative Deep Update (Avoiding Recursion)
  5. Method 3: Using the ** Operator (With Limitations)
  6. Method 4: Leveraging Third-Party Libraries (e.g., deepmerge)
  7. Practical Use Cases
  8. Handling Edge Cases
  9. Conclusion
  10. References

1. Understanding Nested Dictionaries#

A nested dictionary is a dictionary where one or more values are themselves dictionaries. This creates a tree-like structure. For example:

# A nested dictionary representing user data
user_data = {
    "name": "Alice",
    "preferences": {
        "theme": "light",
        "notifications": {
            "email": True,
            "push": False
        }
    },
    "metadata": {"version": 1.0}
}

Here, preferences and notifications are nested dictionaries. Updating user_data naively (e.g., overwriting preferences) would erase existing keys like notifications.email.

2. The Pitfall of Shallow Updates#

Python’s built-in dict.update() method and the ** operator perform shallow updates, meaning they overwrite entire nested dictionaries instead of merging them.

Example of Shallow Update Failure:#

user_data = {
    "preferences": {
        "theme": "light",
        "notifications": {"email": True}
    }
}
 
# Attempt to update only "theme" (shallow update)
update = {"preferences": {"theme": "dark"}}
user_data.update(update)
 
print(user_data["preferences"])  
# Output: {'theme': 'dark'} → "notifications" is LOST!

The update() method overwrites the entire preferences dictionary, erasing notifications. To avoid this, we need deep updates that recursively merge nested dictionaries.

3. Method 1: Recursive Deep Update#

The most straightforward way to update nested dictionaries without overwriting is to use a recursive function. This function will:

  1. Iterate over keys in the update dictionary.
  2. If a key exists in the original dictionary and both values are dictionaries, recursively update the nested dictionary.
  3. Otherwise, set the key to the new value (overwriting only if the value isn’t a nested dictionary).

Step-by-Step Implementation#

def deep_update_recursive(original: dict, update: dict) -> dict:
    """Recursively update a nested dictionary without overwriting existing keys."""
    for key, value in update.items():
        # Check if both original and update values are dictionaries
        if isinstance(value, dict) and key in original and isinstance(original[key], dict):
            # Recursively update the nested dictionary
            deep_update_recursive(original[key], value)
        else:
            # Overwrite or set the value (non-dict or new key)
            original[key] = value
    return original

Example Usage#

# Original nested dictionary
original = {
    "user": {
        "name": "Alice",
        "settings": {
            "theme": "light",
            "notifications": {"email": True}
        }
    },
    "version": 1
}
 
# Dictionary with updates (nested and new keys)
update = {
    "user": {
        "settings": {"theme": "dark"},  # Update only "theme"
        "age": 30  # Add new key "age"
    },
    "version": 2  # Update top-level "version"
}
 
# Apply deep update
deep_update_recursive(original, update)
 
print(original)

Output:#

{
    "user": {
        "name": "Alice",
        "settings": {
            "theme": "dark",  # Updated
            "notifications": {"email": True}  # Preserved!
        },
        "age": 30  # New key added
    },
    "version": 2  # Updated
}

Why it works: The function recursively dives into nested dictionaries (e.g., settings), merging updates instead of overwriting the entire sub-dictionary.

4. Method 2: Iterative Deep Update (Avoiding Recursion)#

Recursion works well for moderately nested dictionaries, but for extremely deep structures (e.g., 1000+ levels), it may hit Python’s recursion limit. An iterative approach using a stack/queue avoids this by simulating recursion with a loop.

Implementation#

def deep_update_iterative(original: dict, update: dict) -> dict:
    """Iteratively update a nested dictionary using a stack (avoids recursion)."""
    stack = [(original, update)]  # Stack stores (original_subdict, update_subdict)
    
    while stack:
        current_original, current_update = stack.pop()
        
        for key, value in current_update.items():
            # If both values are dictionaries, push nested dicts to stack
            if isinstance(value, dict) and key in current_original and isinstance(current_original[key], dict):
                stack.append((current_original[key], value))
            else:
                current_original[key] = value  # Overwrite or set new key
    return original

Example Usage#

Using the same original and update dictionaries as in Method 1:

deep_update_iterative(original, update)
print(original)  # Same output as recursive method!

Advantage: Safe for deeply nested dictionaries (no recursion depth issues).

5. Method 3: Using the ** Operator (With Limitations)#

The ** operator merges dictionaries but only at the shallow level. To use it for nested updates, combine it with the recursive/iterative functions above for a concise syntax.

Example: Shallow Merge Limitation#

original = {"a": 1, "b": {"x": 10}}
update = {"b": {"y": 20}, "c": 3}
 
# Shallow merge with ** operator
shallow_merged = {**original,** update}
print(shallow_merged["b"])  # Output: {'y': 20} → "x" is lost!

Fix with ** + Deep Update#

To merge top-level keys and deep-merge nested ones:

# Create a copy of original, then deep-update with the update dict
merged = deep_update_recursive({**original}, update)
print(merged["b"])  # Output: {'x': 10, 'y': 20} (preserved and updated!)

Here, {**original} creates a shallow copy of original, and deep_update_recursive merges update into it, preserving nested keys.

6. Method 4: Leveraging Third-Party Libraries (e.g., deepmerge)#

For complex use cases (e.g., merging lists, custom conflict resolution), use battle-tested libraries like deepmerge instead of writing custom code.

Step 1: Install deepmerge#

pip install deepmerge

Step 2: Merge Dictionaries#

deepmerge provides a Merger class with flexible merging logic. By default, it merges dictionaries and overrides non-dict values.

from deepmerge import Merger
 
# Configure merger to merge dicts and override other types
merger = Merger(
    [(dict, ["merge"])],  # Merge dictionaries
    ["override"],         # Override non-dict values (e.g., int, str)
    ["override"]          # Override lists (default behavior)
)
 
original = {
    "preferences": {"theme": "light", "notifications": {"email": True}}
}
update = {"preferences": {"theme": "dark", "notifications": {"push": True}}}
 
# Merge the dictionaries
result = merger.merge(original, update)
print(result["preferences"])

Output:#

{
    "theme": "dark", 
    "notifications": {"email": True, "push": True}  # Both keys preserved!
}

Advantages:

  • Handles edge cases (e.g., merging lists with ["append"] strategy).
  • Tested for performance and reliability.

7. Practical Use Cases#

Use Case 1: Merging Default and User Configs#

Applications often use default settings with optional user overrides. Deep updates ensure user settings don’t erase defaults:

default_config = {
    "appearance": {"theme": "light", "font_size": 12},
    "notifications": {"email": True}
}
 
user_config = {"appearance": {"theme": "dark"}, "notifications": {"push": True}}
 
# Merge user_config into default_config
final_config = deep_update_recursive(default_config.copy(), user_config)
print(final_config)

Output:#

{
    "appearance": {"theme": "dark", "font_size": 12},  # "font_size" preserved
    "notifications": {"email": True, "push": True}      # Both email/push
}

Use Case 2: Updating JSON Data from APIs#

When updating a JSON response with new data, deep merging preserves existing fields:

api_response = {
    "user": {"id": 123, "details": {"name": "Alice", "hobbies": ["reading"]}}
}
 
new_data = {"user": {"details": {"hobbies": ["coding"]}, "status": "active"}}
 
deep_update_recursive(api_response, new_data)
print(api_response["user"]["details"]["hobbies"])  # Output: ['coding'] (overridden list)
print(api_response["user"]["status"])              # Output: 'active' (new key)

8. Handling Edge Cases#

Edge Case 1: Non-Dict Values Overwriting Dicts#

If the update contains a non-dict value where the original has a dict, the update will overwrite the entire nested dict (intended behavior):

original = {"a": {"b": 1}}
update = {"a": 2}  # Non-dict value
deep_update_recursive(original, update)
print(original["a"])  # Output: 2 (overwritten)

Edge Case 2: Lists Are Not Merged#

Deep update methods merge only dictionaries. Lists are treated as atomic values and overwritten:

original = {"a": [1, 2]}
update = {"a": [3]}
deep_update_recursive(original, update)
print(original["a"])  # Output: [3] (list overwritten)

To merge lists, use deepmerge with the ["append"] strategy for lists.

9. Conclusion#

Updating nested dictionaries without overwriting requires deep merging rather than shallow updates. Choose the method that fits your needs:

  • Custom recursive/iterative functions: Best for simple to moderate cases where you need full control.
  • Libraries like deepmerge: Ideal for complex scenarios (e.g., merging lists, custom conflict rules) or production code.

By avoiding shallow updates, you’ll preserve critical data and build more robust applications.

10. References#

2025-11