diff --git a/VERSION b/VERSION index de70df709d97e020f7d2c2d910da3827b8221597..e7fa0fa57ea09b5e918333e5eacc159e85e635f4 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -20.3 \ No newline at end of file +20.4 \ No newline at end of file diff --git a/docs/src/en/releases/gen-20.4.rst b/docs/src/en/releases/gen-20.4.rst new file mode 100644 index 0000000000000000000000000000000000000000..e10d7d0988755d2d929da6f2229641b86ea1991c --- /dev/null +++ b/docs/src/en/releases/gen-20.4.rst @@ -0,0 +1,10 @@ +.. _release-gen-20.4: + +=============== +20.4 Generation +=============== + +.. toctree:: + :titlesonly: + + /releases/release-20.4 diff --git a/docs/src/en/releases/noc-20.rst b/docs/src/en/releases/noc-20.rst index ab7dfcfb8640f4972ae2031528d86ad7370796fd..741577f483eb8bc597528e520dad9d6de37c77f0 100644 --- a/docs/src/en/releases/noc-20.rst +++ b/docs/src/en/releases/noc-20.rst @@ -7,6 +7,7 @@ .. toctree:: :titlesonly: + /releases/gen-20.4 /releases/gen-20.3 /releases/gen-20.2 /releases/gen-20.1 diff --git a/docs/src/en/releases/release-20.4.rst b/docs/src/en/releases/release-20.4.rst new file mode 100644 index 0000000000000000000000000000000000000000..c4ec8671bfe51ee89819a063e087436d27acff0f --- /dev/null +++ b/docs/src/en/releases/release-20.4.rst @@ -0,0 +1,817 @@ +.. _release-20.4: + +================ +NOC 20.4 +================ + +20.4 release contains `225 `_ bugfixes, optimisations and improvements. + +.. _release-20.4-highlights: +Highlights +---------- + +.. _release-20.4-gmx: +Generic Message Exchange +^^^^^^^^^^^^^^^^^^^^^^^^ +NOC can send notifications to email/telegram via Notification groups +on alarms and configuration changes. Notifications are useful +to take human attention to possible problem. To notify push data +to external system NOC uses DataStream approach. External systems +have to pull changes and process them according own logic. + +NOC 20.4 generalises all data pushed to external systems to the +concepts of `messages`. Message is the piece of data which +can be passed from NOC to outside. Messages can be of different types: + +* alarms +* object inventory data +* configuration +* configuration change +* reboot +* new object +* system login +* etc. + +NOC can generate messages on certain condition. Humans and soulless +robots can have interest in messages. So we need some kind of routing. + +NOC 20.4 introduces new service, called `Message Exchanger` or `mx`. +Like mail servers, `mx` receives the message, processes it headers +and decides where to route the message. `mx` relies on family of the +`sender` processes. Each kind of sender can deliver the message +outside of the system. Each sender supports particular exchange protocol, +hiding delivery details from `mx`. `mx` can transform messages or apply +the templates to convert delivered message to desired format. +`mx`, `senders`, message generation and transport conventions became +the viable part of NOC called `Generalized Message Exchange` or `GMX`. + +NOC 20.4 introduces `kafkasender` service, used to push data to a Kafka +message bus. We're planning to convert other senders (`mailsender`, `tgsender`, etc) +to GMX in the NOC 21.1. + +Kafka Integration +^^^^^^^^^^^^^^^^^ +NOC 20.4 introduces the `kafkasender` service, the part of GMX. Kafka +became mainstream message bus in telecom operation, and NOC is +being able to push all data, available via DataStream to a Kafka +for following routing and processing, reducing amount of mutual +system-to-system integrations. + + +.. _release-20.4-biosegmentation: +Biosegmentation +^^^^^^^^^^^^^^^ +Biosegmentation has been introduced in NOC 20.3 as ad-hoc segmentation process. +Process relies on the series of `trials`. Each trial can lead to merging +or fixing the structure of segments tree. Current implementation +relies on inter-segment links. But sometimes the segment hierarchy +must be established before the linking process. + +NOC 20.4 introduces additional MAC-based biosegmentation approach, +called `Vacuum Bulling`, allowing to build segment hierarchy basing +on MAC addresses, collected on interfaces. + +.. _release-20.4-lift: +Ordered Message Queue +^^^^^^^^^^^^^^^^^^^^^ +NOC uses NSQ as internal message queue. Lightweight and hi-performance +solution shows good result usually. But after the time architectural corner +cases became more and more visible: + +* NSQ designed to be always-on-dial solution. nsqd is on every host, + communicating to publisher via localhost loopback. In modern container + world that fact being bug, not a feature. Reliance on absolute reliability + of connection between publisher and broker became unacceptable. +* Subscribers have to communicate with nsqlookup service to find the hosts containing data. + Then they have to establish direct connection with them. + Official python NSQ client uses up to 5 tcp connections. + So amount of connection grows fast with grow of amount producers and subscribers. +* Official python NSQ client's error handling is far from ideal. + Code base is old and obscure and hard to maintain. No asyncio version + is available. +* No fault tolerance. Failed nsqd will lead to the lost messages. + No message replication at all. +* Out-of-order messages. Message order may change due to internal nsqd + implementation and to client logic. Applications like fault management + relies on message order. Closing events must follow opening ones. + Otherwise the hanging alarms will pollute the system. + +During the researches we'd decided we need message system with commit-log approach. +Though Kafka is industrial standard, its dependency on JVM and Zookeeper may be a burden. +We stopped on Liftbridge. Liftbridge is clean and simple implementation of proven Kafka +storage and replication algorithms. + +We'd ported `events` topics to Liftbridge, fixing critical events ordering +problem. GMX topics uses Liftbridge too. Next release (21.1) will address remaining topics. + +.. _release-20.4-fastapi: +FastAPI +^^^^^^^ +We'd starting migration from Tornado to FastAPI. Main motivation is: + +* Tornado has bring generator-based asynchronous programming to Python2. + Python3 has introduced native asynchronous programming along with asyncio library. + Later Tornado versions are simple wrappers atop asyncio. +* FastAPI uses Pydantic for request and response validation. + We'd considered Pydantic very useful during out ETL refactoring +* FastAPI generates OpenAPI/Swagger scheme, improving integration capabilities. +* FastAPI is fast. + +We'd ported `login` service to FastAPI. JWT had replaced Tornado's signed cookies. +We'd also implemented the set of OAuth2-based endpoints for our next-generation UI. + +.. _release-20.4-etl: +ETL Improvements +^^^^^^^^^^^^^^^^ +ETL has relied on CSV format to store extracted data. +Though it simple and wraps SQL responses in obvious way, it have some limitation: + +* Metadata of extracted fields stored outside of extractor, in the loader. +* Field order hardcoded in loader +* Fields has no type information, leading to leaky validation +* No native way to pass complex data structures, like list and nested documents +* Extractors must return empty data for long time deprecated fields + +NOC 20.4 introduces new extractor API. Instead of lists, passed to CSV, +extractor returns pydantic model instances. Pydantic models +are defined in separate modules and reused by both extractors and loaders. +Interface between extractor and loader became well-defined. +Models perform data validation on extraction and load stages. +So errors in extractor will lead to informative error message and to the stopping of process. + +ETL now uses JSON Line format (jsonl) - a bunch of JSON structures for each row, separated by newlines. So it is possible +to store structures with arbitrary complexity. +We'd ever provided the tool to convert legacy extracted data to a new format. + +.. _release-20.4-snmp-ratelimit: +SNMP Rate Limiting +^^^^^^^^^^^^^^^^^^ +NOC 20.4 allows to limit a rate of SNMP requests basing on profile +or platform settings. This reduces impact on the platforms with weak +CPU or slow control-to-dataplane bus. + +.. _release-20.4-orjson: +orjson +^^^^^^ +orjson is used instead of ujson for JSON serialization/deserialization. + +.. _release-20.4-new-profiles: +New profiles +^^^^^^^^^^^^ + +* KUB Nano +* Qtech.QFC + +.. _release-20.4-migration: +Migration +--------- + +Tower Upgrade +^^^^^^^^^^^^^ +Please upgrade Tower up to 1.0.0 or later before continuing NOC +installation/upgrade process. +See `Tower upgrade process documentation`_ +for more details. + +Elder versions of Tower will stop deploy with following error message + +Liftbridge/NATS +^^^^^^^^^^^^^^^ +NOC 20.4 introduces Liftbridge service for ordered message queue. +You should deploy at least 1 Liftbridge and 1 NATS service instance. +See more details in Tower's service configuration section. + +ETL +^^^ +Run fix after upgrade + + ./noc fix apply fix_etl_jsonl + +.. _release-20.4-features: +New features +------------ ++------------+-------------------------------------------------------------------------+ +| MR | Title | ++------------+-------------------------------------------------------------------------+ +| :mr:`1668` | Added function get alarms for controllers and devices for periodic job. | ++------------+-------------------------------------------------------------------------+ +| :mr:`4223` | FastAPI login service | ++------------+-------------------------------------------------------------------------+ +| :mr:`4256` | Add Project to ETL | ++------------+-------------------------------------------------------------------------+ +| :mr:`4274` | New profile Qtech.QFC | ++------------+-------------------------------------------------------------------------+ +| :mr:`4290` | Liftbridge client | ++------------+-------------------------------------------------------------------------+ +| :mr:`4361` | #1363 ifdesc: Interface autocreation | ++------------+-------------------------------------------------------------------------+ +| :mr:`4388` | Add new controller profile KUB Nano | ++------------+-------------------------------------------------------------------------+ +| :mr:`4398` | mx service | ++------------+-------------------------------------------------------------------------+ +| :mr:`4403` | kafkasender service | ++------------+-------------------------------------------------------------------------+ +| :mr:`4473` | #1368 Model Interface scopes | ++------------+-------------------------------------------------------------------------+ +| :mr:`4488` | #892 ETL JSON format | ++------------+-------------------------------------------------------------------------+ +| :mr:`4519` | noc/noc#1356 SNMP Rate Limit | ++------------+-------------------------------------------------------------------------+ +| :mr:`4538` | Configurable LDAP server policies | ++------------+-------------------------------------------------------------------------+ +| :mr:`4567` | Biosegmentation: Vacuum bulling | ++------------+-------------------------------------------------------------------------+ + + +.. _release-20.4-improvements: +Improvements +------------ ++------------+----------------------------------------------------------------------------------------------+ +| MR | Title | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4225` | Fix ddash refid | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4233` | Allow alternative locations for binary speedup modules | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4236` | Catch when sentry-sdk module enabled but not installed. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4246` | Fix Qtech.BFC profile | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4261` | noc/noc#1304 Replace ujson with orjson | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4264` | runtime optimization ReportMaxMetrics | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4275` | ElectronR.KO01M profile scripts | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4278` | noc/noc#1383 Add IfPath collator to confdb | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4280` | noc/noc#1381 Add alarm_consequence_policy to TTSystem settings. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4281` | #1384 Add source-ip aaa hints. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4287` | Add round argument to metric scale function | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4293` | Debian-based docker image | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4296` | Change python to python3 when use `./noc` | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4314` | Update Card for Sensor Controller | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4320` | Fill capabilities for beef. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4338` | New Grafana dashboards | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4344` | Profile fix controllers | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4348` | `exp_decay` window function | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4349` | Controller/fix2 | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4354` | add_interface-type_Juniper_JUNOSe | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4358` | Fix Qtech.BFC profile | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4364` | LiftBridgeClient: Proper handling of message headers | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4369` | LiftBridgeClient: fetch_metadata() `stream` and `wait_for_stream` parameters | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4380` | Add to_json for thresholdprofile | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4383` | Update threshold handler | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4384` | Add collators to some profiles. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4389` | Electron fix profile | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4391` | add new metric Qtech.BFC | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4394` | fix some controllers ddash/metrics | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4396` | Fix inerfaces name Qtech.BFC | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4399` | Up report MAX_ITERATOR to 800 000. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4402` | mx: Use FastAPIService | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4405` | liftbridge cursor persistence api | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4407` | add_columns_total_reportmaxmetrics | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4416` | Add csv+zip format to ReportDetails. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4417` | Add `Long Alarm Archive` options to ReportAlarm, from Clickhouse table. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4428` | Add available_only options to ReportDiscoveryTopologyProblem. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4432` | Reset NetworkSegment TTL cache after remove. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4433` | Change is_uplink criterias priority on segment MAC discovery. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4439` | fix_reportmaxmetrics | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4447` | Add octets_in_sum and octets_out_sum columns to ReportMetrics. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4453` | ConfDB syslog | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4455` | Fix controllers profiles, ddash | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4457` | Fix get_iface_metrics | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4462` | noc/noc#1392 Add search port by contains ifdescription token to ifdecr discovery. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4464` | LiftBridge client: Connection pooling | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4470` | Add ReportMovedMacApplication application. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4475` | Add sorted to tags application. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4477` | noc/noc#1416 Extend ConfDB meta section. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4479` | Add get_confdb_query method to ManagedObjectSelector and MatchPrefix ConfDB function. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4480` | Add csv_zip file format to MetricsDetail Report. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4483` | noc/noc#1397 Additional biosegtrial criteria to policy. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4486` | Add migrate_ts field to ReportMovedMac. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4501` | noc/noc#1428 Add InterfaceDiscoveryApplicator for fill ConfDB info from interface discovery. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4508` | add_csvzip_reportmaxmetrics | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4511` | Fix ./noc discovery for LB | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4515` | noc/noc#1432 lb client: Configurable message size limit | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4516` | fix csv_import view | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4517` | Additional options to segment command | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4535` | Bump networkx/numpy requirements | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4539` | lb client: increased resilience | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4547` | Add JOB_CLASS param to core.defer util. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4549` | ETL model Reference | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4551` | add column reboots in fm.reportalarmdetail | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4553` | fix processing trunk port vlan for HP A3100-24 (v 5.20.99) | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4565` | Add ttl-policy argument to link command. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4571` | Filter Multicast MACs on Moved MAC report. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4573` | Add api_unlimited_row_limit param | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4579` | liftBridge: publish_async waits for all the acks | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4582` | noc/noc#1371 Add schedule_discovery_config handler to events.discovery. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4592` | noc/noc#1400 Migrate InterfaceClassification to ConfDB. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4602` | Add MatchAllVLAN and MatchAnyVLAN function to ConfDB. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4607` | Bump pytest version | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4624` | add metrics "Subscribers | Summary" Alcatel.TIMOS | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4629` | noc/noc#1440 Use all macs on 'Discovery ID cache poison' report. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4630` | Convert limit from dcs to int. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4632` | Add Telephony SIP metrics graph. | ++------------+----------------------------------------------------------------------------------------------+ +| :mr:`4633` | Always uplinks calculate. | ++------------+----------------------------------------------------------------------------------------------+ + + +.. _release-20.4-bugs: +Bugfixes +-------- ++------------+-------------------------------------------------------------------------------------------+ +| MR | Title | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4249` | Fix card MO | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4251` | Fix status RNR | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4258` | Change field_num on ReportObjectStat | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4269` | noc/noc#1374 Fix typo on datastream format check. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4285` | Fix Profile Check Summary typo. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4303` | #1335 ConfDB: Fix `and` inside `or` combination | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4310` | Fix RNR affected AD | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4319` | Add err_status to beef snmp_getbulk_response method. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4321` | Convert oid on snmp raw_varbinds. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4322` | Fix event clean | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4327` | Convert set to list on orjson dumps. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4328` | Add xmac discovery to ReportDiscoveryResult. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4363` | ./noc migrate-liftbridge: Do not create streams for disabled services | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4368` | Fix hash_int() | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4373` | Fix typo on Calcify Biosegmentation policy. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4409` | Add get_pool_partitions method to TrapCollectorService. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4418` | Add id field to project etl loader. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4419` | Fix multiple segment args on discovery command. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4423` | noc/noc#1399 Delete Permissions and Favorites on wipe user. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4424` | noc/noc#1375 Fix DEFAULT_STENCIL use on SegmentTopology. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4425` | noc/noc#1396 AlarmEscalation. Use item delay for consequence escalation. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4426` | Fix extapp group regex splitter to non-greedy. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4430` | Fix ManagedObject _reset_caches key for _id_cache. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4452` | noc/noc#1406 Use system username for JWT. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4461` | noc/noc#1229 Fix user cleanup Django Admin Log. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4472` | Add audience param to is_logged jwt.decode. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4474` | Add 120 sec to out_of_order escalation time. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4485` | noc/noc#688 Fix invalidate l1 cache for ManagedObject. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4492` | Skipping files if already compressed on destination. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4497` | noc/noc#1427 Fix whois ARIN url. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4498` | Fix object data use. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4502` | Move orjson defaults to jsonutils. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4505` | Bump ssh2-python to 0.23. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4506` | pm/utils -> Fix dict | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4507` | Some etl loader fixes. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4513` | noc/noc#1423 Convert pubkey to bytes. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4514` | Convert empty object data to list on 0020 migration. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4518` | Fix vendors and handlers migrations | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4522` | Fix typo on ifdescr discovery. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4524` | #1312 Consistent VPN ID generation | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4540` | Fix customfields for mongoengine. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4555` | Revert uvicorn to 0.12.1. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4561` | Fix typo on interfaceprofile UI Application. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4564` | Fix trace when execute other script that command on MRT. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4569` | Fix typo on MRT service. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4575` | Add static_service_groups and static_client_groups clean_map to managedobject etl loader. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4590` | Fix login cookie ttl | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4594` | Fix ETL loader change. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4595` | Fix extra filter when set extra order. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4598` | Fix datetime field on Service ETL model. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4614` | Fix SNMP_GET_OIDS on get_chassis_id scripts to list. | ++------------+-------------------------------------------------------------------------------------------+ +| :mr:`4627` | noc/noc#1439 Fix tag contains query for non latin symbol. | ++------------+-------------------------------------------------------------------------------------------+ + + +.. _release-20.4-cleanup: +Code Cleanup +------------ ++------------+-----------------------------------------------------------------------------------------------+ +| MR | Title | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4254` | Cleanup flake. | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4301` | Fix vendor docs test | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4317` | Updated .dockerignore | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4360` | Remove unused dependencies: tornadis, mistune | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4362` | Update blinker, bsdiff, cachetools, crontab, progressbar2, psycopg2, python-dateutil versions | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4465` | Remove legacy scripts/ci-run | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4496` | Fix formatting | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4533` | Bump requirements | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4587` | Fix collect beef for orjson. | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4589` | Fix some lint errors | ++------------+-----------------------------------------------------------------------------------------------+ +| :mr:`4622` | Fix Service etl model. | ++------------+-----------------------------------------------------------------------------------------------+ + + +.. _release-20.4-profiles: +Profile Changes +--------------- + +.. _release-20.4-profile-Cisco.IOS: +Cisco.IOS +^^^^^^^^^ ++------------+--------------------------------------------------------------+ +| MR | Title | ++------------+--------------------------------------------------------------+ +| :mr:`4316` | Update Cisco.IOS profile to support more physical interfaces | ++------------+--------------------------------------------------------------+ + + +.. _release-20.4-profile-Cisco.IOSXR: +Cisco.IOSXR +^^^^^^^^^^^ ++------------+-----------------------------------------+ +| MR | Title | ++------------+-----------------------------------------+ +| :mr:`4408` | added interfacetypes for IOSXR platform | ++------------+-----------------------------------------+ + + +.. _release-20.4-profile-DLink.DxS: +DLink.DxS +^^^^^^^^^ ++------------+------------------------------------------------------------------+ +| MR | Title | ++------------+------------------------------------------------------------------+ +| :mr:`4355` | DLink.DxS.get_metrics. Fix SNMP Error when 'CPU | Usage' metric. | ++------------+------------------------------------------------------------------+ +| :mr:`4434` | Fix Dlink.DxS profile. | ++------------+------------------------------------------------------------------+ + + +.. _release-20.4-profile-EdgeCore.ES: +EdgeCore.ES +^^^^^^^^^^^ ++------------+-------------------------------------------------------------------------+ +| MR | Title | ++------------+-------------------------------------------------------------------------+ +| :mr:`4556` | EdgeCore.ES.get_spanning_tree. Fix getting port_id for Trunk interface. | ++------------+-------------------------------------------------------------------------+ + + +.. _release-20.4-profile-Eltex.MES: +Eltex.MES +^^^^^^^^^ ++------------+-------------------------------------------------------------------------+ +| MR | Title | ++------------+-------------------------------------------------------------------------+ +| :mr:`4217` | test tacacs1.yml crashed. AssertionError: assert [] == [(right syntax)] | ++------------+-------------------------------------------------------------------------+ +| :mr:`4262` | Eltex.MES.get_capabilities. Fix detect stack mode by SNMP. | ++------------+-------------------------------------------------------------------------+ +| :mr:`4523` | Eltex.MES.get_vlans. Use Generic script. | ++------------+-------------------------------------------------------------------------+ +| :mr:`4615` | Eltex.MES. Add 1.3.6.1.4.1.89.53.4.1.7.1 to display_snmp. | ++------------+-------------------------------------------------------------------------+ + + +.. _release-20.4-profile-Eltex.MES24xx: +Eltex.MES24xx +^^^^^^^^^^^^^ ++------------+-----------------------------------------+ +| MR | Title | ++------------+-----------------------------------------+ +| :mr:`4381` | Fix Eltex.MES24xx.get_interfaces script | ++------------+-----------------------------------------+ + + +.. _release-20.4-profile-Extreme.XOS: +Extreme.XOS +^^^^^^^^^^^ ++------------+-------------------------------------------+ +| MR | Title | ++------------+-------------------------------------------+ +| :mr:`4404` | Fix Extreme.XOS.get_lldp_neighbors script | ++------------+-------------------------------------------+ + + +.. _release-20.4-profile-Generic: +Generic +^^^^^^^ ++------------+--------------------------------------------------------------------------------+ +| MR | Title | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4239` | Generic.get_capabilities add 'SNMP | OID | EnterpriseID' len check. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4342` | Generic.get_arp. Cleanup snmp for py3 | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4613` | Generic.get_chassis_id. Add 'LLDP-MIB::lldpLocChassisId' oid to display_hints. | ++------------+--------------------------------------------------------------------------------+ + + +.. _release-20.4-profile-Huawei.MA5600T: +Huawei.MA5600T +^^^^^^^^^^^^^^ ++------------+-------------------------------------------------------+ +| MR | Title | ++------------+-------------------------------------------------------+ +| :mr:`4611` | Huawei.MA5600T.get_spanning_tree. Fix waited command. | ++------------+-------------------------------------------------------+ + + +.. _release-20.4-profile-Huawei.VRP: +Huawei.VRP +^^^^^^^^^^ ++------------+------------------------------------------------------------+ +| MR | Title | ++------------+------------------------------------------------------------+ +| :mr:`4422` | Huawei.VRP. Add NE8000 version detect. | ++------------+------------------------------------------------------------+ +| :mr:`4550` | Huawei.VRP fix normalize_enable_stp | ++------------+------------------------------------------------------------+ +| :mr:`4557` | Huawei.VRP. Check nexthop type on ConfDB route normalizer. | ++------------+------------------------------------------------------------+ + + +.. _release-20.4-profile-Juniper.JUNOS: +Juniper.JUNOS +^^^^^^^^^^^^^ ++------------+-----------------------------------------+ +| MR | Title | ++------------+-----------------------------------------+ +| :mr:`4324` | Fix Juniper.JUNOS.get_chassis_id script | ++------------+-----------------------------------------+ +| :mr:`4377` | Fix Juniper.JUNOS.get_interfaces script | ++------------+-----------------------------------------+ + + +.. _release-20.4-profile-NAG.SNR: +NAG.SNR +^^^^^^^ ++------------+---------------------------------------+ +| MR | Title | ++------------+---------------------------------------+ +| :mr:`4351` | Fix NAG.SNR.get_interfaces script | ++------------+---------------------------------------+ +| :mr:`4481` | Fix NAG.SNR.get_lldp_neighbors script | ++------------+---------------------------------------+ + + +.. _release-20.4-profile-Qtech.QSW: +Qtech.QSW +^^^^^^^^^ ++------------+-----------------------+ +| MR | Title | ++------------+-----------------------+ +| :mr:`4576` | Fix Qtech.QSW profile | ++------------+-----------------------+ + + +.. _release-20.4-profile-Qtech.QSW2800: +Qtech.QSW2800 +^^^^^^^^^^^^^ ++------------+--------------------------------------+ +| MR | Title | ++------------+--------------------------------------+ +| :mr:`4444` | Qtech.QSW2800. Add sdiag prompt. | ++------------+--------------------------------------+ +| :mr:`4542` | Fix Qtech.QSW2800.get_version script | ++------------+--------------------------------------+ + + +.. _release-20.4-profile-Ubiquiti.AirOS: +Ubiquiti.AirOS +^^^^^^^^^^^^^^ ++------------+----------------------------------------------+ +| MR | Title | ++------------+----------------------------------------------+ +| :mr:`4240` | Ubiquiti.AirOS.get_version. Cleanup for py3. | ++------------+----------------------------------------------+ + + +.. _release-20.4-profile-rare: +rare +^^^^ ++------------+--------------------------------------------------------------------------------+ +| MR | Title | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4214` | ConfDB tests profile Raisecom.RCIOS. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4241` | Alstec.MSPU.get_version. Fix HappyBaby platform regex. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4265` | Fix ZTE.ZXA10 profile | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4272` | Eltex.WOPLR. Add get_interface_type method to profile. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4279` | Update Rotek.BT profile | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4288` | Add Enterasys.EOS profile | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4295` | Fix metric name | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4302` | add snmp in profile Juniper.JUNOSe | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4313` | Rotek.BT fix get_metrics | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4335` | add snmp in profile Alcatel.TIMOS | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4353` | Update ZTE.ZXA10 profile to support C610 | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4365` | Fix prompt matching in Fortinet.Fortigate profile | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4371` | Alcatel.OS62xx.get_version. Set always_prefer to S for better platform detect. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4376` | fix_get_lldp_neighbors_NSN.TIMOS | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4406` | Add AcmePacket.NetNet profile. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4431` | noc/noc#1391 Cisco.WLC. Add get_interface_type method. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4536` | add_bras_metrics_Juniper_JUNOSe | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4570` | Fix h3c get_switchport | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4578` | Eltex.ESR add snmp support | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4583` | Update DCN.DCWS profile.py | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4585` | Update sa/profiles/DCN/DCWS/get_config.py | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4586` | Ericsson.SEOS.get_interfaces. Migrate to Generic SNMP. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4596` | Fix DLink.DxS_Smart profile | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4600` | Huawei.VRP3.get_interface_status_ex. Fix return in/out speed as kbit/sec. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4610` | Huawei.VRP3.get_interface_status_ex. Fix trace when SNMP Timeout. | ++------------+--------------------------------------------------------------------------------+ +| :mr:`4617` | NSN.TIMOS.get_interfaces. Fix empty MAC on output. | ++------------+--------------------------------------------------------------------------------+ + + +.. _release-20.4-collections: +Collections Changes +------------------- ++------------+-----------------------------------------------------------+ +| MR | Title | ++------------+-----------------------------------------------------------+ +| :mr:`4277` | Add more Juniper part number | ++------------+-----------------------------------------------------------+ +| :mr:`4282` | Add new caps - Sensor | Controller | ++------------+-----------------------------------------------------------+ +| :mr:`4294` | New Environment metrics | ++------------+-----------------------------------------------------------+ +| :mr:`4305` | Fix bad json on collection. | ++------------+-----------------------------------------------------------+ +| :mr:`4307` | Cleanup HP fm.eventclassificationrule. | ++------------+-----------------------------------------------------------+ +| :mr:`4337` | Fix get metrics script for controller | ++------------+-----------------------------------------------------------+ +| :mr:`4345` | Fix dev.specs SNMP chassis for Huawei and Generic. | ++------------+-----------------------------------------------------------+ +| :mr:`4411` | Add some Juniper models | ++------------+-----------------------------------------------------------+ +| :mr:`4451` | Add some Juniper models | ++------------+-----------------------------------------------------------+ +| :mr:`4460` | noc/noc#1411 Add PhonePeer MetricScope. | ++------------+-----------------------------------------------------------+ +| :mr:`4499` | Fix default username BI dashboard. | ++------------+-----------------------------------------------------------+ +| :mr:`4520` | sa.profilecheckrules: Eltex | MES | MES5448 sysObjectID.0 | ++------------+-----------------------------------------------------------+ +| :mr:`4625` | Add AcmePacket Vendor. | ++------------+-----------------------------------------------------------+ + + +.. _release-20.4-deploy: +Deploy Changes +-------------- ++------------+-----------------------------------------------------+ +| MR | Title | ++------------+-----------------------------------------------------+ +| :mr:`4478` | noc/noc#1241 Merge ansible deploy to master repo | ++------------+-----------------------------------------------------+ +| :mr:`4623` | Add liftbridge deployflow | ++------------+-----------------------------------------------------+ +| :mr:`4637` | Fix auth path redirect | ++------------+-----------------------------------------------------+ +| :mr:`4640` | Catch trace on etl loader when delete lost mapping. | ++------------+-----------------------------------------------------+ +| :mr:`4643` | Change start condition | ++------------+-----------------------------------------------------+