Migrate to mongoengine 0.13/pymongo 3.5.1

Known pymongo incompatibilities:

aggregate

PyMongo 2.6 added an option to return an iterable cursor from aggregate(). In PyMongo 3 aggregate() always returns a cursor

r["ok"] checks must be removed
for record in r["result"] must be replaced with for record in r

Affected modules:

cursor timeouts

find(...., timeout=False) must be replaced with find(..., no_cursor_timeout=True)

Affected modules:

core/etl/bi/extractor/alarms.py
core/etl/bi/extractor/reboots.py

pymongo connection options

The “slave_okay” option is removed The slave_okay option is removed from PyMongo’s API. The secondaryPreferred read preference provides the same behavior. Code like this:

client = MongoClient(slave_okay=True)

can be changed to this with PyMongo 2.9 or newer:

client = MongoClient(readPreference="secondaryPreferred")

lib/nosql.py

MongoEngine connection error handling

ConnectionError is MongoEngineConnectionError now

lib/nosql.py

safe option is removed

In PyMongo 3 the safe option is removed from the entire API. MongoClient has always defaulted to acknowledged write operations and continues to do so in PyMongo 3.

core/scheduler/scheduler.py
gis/migrations/0002_world_area.py

use update_many instead of multi=True

Use update_many for multi-document updates

update operations return UpdateResult

r["ok"] must be replaced with r.acknowledges

r["nModified"] must be replaced with r.modified_count

core/scheduler/scheduler.py
fm/models/activealarm.py

BulkWrite API changes

initialize_ordered_bulk_op/initialize_unordered_bulk_op should be replaced with simpler bulk_write API

from pprint import pprint
from pymongo import InsertOne, DeleteMany, ReplaceOne, UpdateOne
result = db.test.bulk_write([
    DeleteMany({}),  # Remove all documents from the previous example.
    InsertOne({'_id': 1}),
    InsertOne({'_id': 2}),
    InsertOne({'_id': 3}),
    UpdateOne({'_id': 1}, {'$set': {'foo': 'bar'}}),
    UpdateOne({'_id': 4}, {'$inc': {'j': 1}}, upsert=True),
    ReplaceOne({'j': 1}, {'j': 2})])

bulk_write operations return BulkWriteResult instead of dict.

r["nInserted"] must be replaced with r.inserted_count

r["nModified"] must be replaced with r.modified_count

r["nRemoved"] must be replaced with r.deleted_count

The “read_preference” attribute is immutable

Code like this:

cursor = collection.find({"a": 1}, read_preference=ReadPreference.SECONDARY)

can be changed to this with PyMongo 2.9 or later:

coll2 = collection.with_options(read_preference=ReadPreference.SECONDARY)
cursor = coll2.find({"a": 1})

NB:

All changes are performed on feature/moversion branch

Edited Nov 14, 2017 by Andrey Vertiprahov