- Home
- Examples
Examples of our work
Each entry describes a real job: the problem, what we did, the result or the point we have reached. Client names and the details that would make them recognisable have been removed on purpose. The status says whether the work is in production, delivered, in progress, a prototype or a study.
-
A site network extended all the way to the data centre
- The problem
- The LAN and two VLANs of the site had to reach, at layer 2, the servers hosted in the data centre; the previous solution over L2TP and IPsec was not enough.
- What we did
- Three EoIP tunnels over WireGuard, with a usable MTU of 1378 bytes. Two pitfalls solved along the way: the site firewall’s NAT dropped the replies when the WireGuard source port was the same as the destination port, and the allowed addresses in the tunnel had to include all the routed networks.
- Result
- The extension is in production. Automatic high availability between the two site routers was studied and, for now, judged not ready: we wrote that down instead of enabling it.
-
A site connected over 5G with two operators
- The problem
- A site had to work over 5G lines while waiting for fibre, without losing remote management.
- What we did
- Two routers and two 5G lines from different operators: browsing traffic goes out over the second line through policy routing, and the first stays with the tunnels and management. An on-board script saves a restore point after 30 minutes of healthy operation and reloads it if management and the internet go silent. The bandwidth measurement showed that the limiting factor was the router’s CPU (97%), not the line.
- Result
- The site worked over 5G and today 5G acts as backup to the fibre; switching between the two follows a five-step procedure with nine preliminary checks.
-
VPN to the disaster recovery site in the cloud
- The problem
- The disaster recovery site in the cloud had to be reliably reachable, and its configuration was never to put production at risk.
- What we did
- Two tunnels to the two endpoints of the DR site. The script that configures them has preview, verification, rollback and a lock that protects production.
- Result
- The VPN has been tested in both directions on both tunnels (16-18 ms) and manual switchover to DR is ready. Automatic switchover, a three-state machine with hysteresis, is a proposal; having both tunnels active with BGP is a study.
-
Remote access on a firewall without SSL-VPN
- The problem
- That firewall model has no SSL VPN, but users had to work from outside as if they were in the office.
- What we did
- FortiClient over IPsec IKEv2 with EAP: the clients get a LAN address (with proxy ARP) and also reach the VPNs to the other sites without changes at the other end. Behind a hotspot the return traffic was getting lost in the NAT: we forced NAT-T.
- Result
- Remote access is in production; sign-in with company Microsoft Entra ID credentials via SAML is configured and being tested.
-
A VPN server that ran out of memory six times a day
- The problem
- The VPN server used for a client’s backups was being terminated for running out of memory up to six times a day.
- What we did
- The cause was a bridge on a network card that was switched off, which opened one socket per second: 27,233 orphaned sockets, about 7 GB of kernel memory. We removed the bridge and added an automatic check.
- Result
- Memory in use fell from 95% to 3.7%; an automatic check monitors the service.
-
An IPS signature that isolated a server every hour
- The problem
- A server kept losing its connection to the domain at regular intervals, with no obvious trace in the firewall’s traffic logs.
- What we did
- An intrusion prevention signature fired on ONVIF traffic to two cameras, and its block action dropped all of the server’s traffic, 3,700 packets per second; the block renewed itself every time it expired. We created the exception through the APIs of the cloud management system.
- Result
- The server is stable. An automatic check now reads the block tables of five firewall clusters every 3 minutes and sends an alert with an unblock button.
-
Rules that seemed to block but did not
- The problem
- The rules distributed by the cloud management system cannot be read in the local configuration, and the policy match test always returned the same rule.
- What we did
- We used the logs and sessions as our reference, which do not lie, and took an inventory of the services published on the internet. This revealed that the block on malicious addresses was ineffective: an allow rule preceded it by 355 positions.
- Result
- The block was moved ahead of the allow rule; the inventory of published services has been delivered.
-
A site's new network, cut over in a single day
- The problem
- Replacing the firewalls, switches and Wi-Fi of a site in production, together with the VPNs to the cloud and to the other sites.
- What we did
- An active-passive FortiGate cluster, two Aruba CX switches, a UniFi controller and three access points, prepared alongside the existing network. Then five site-to-site VPNs (three route-based to Google Cloud, two to Check Point), remote access and the corporate and guest Wi-Fi.
- Result
- Cutover in a single day, high availability tested, rollback not needed. The next day the check script reported 38 checks passed out of 38; the counters brought to light a faulty port that caused a spanning tree recalculation every time it came back up, and it was shut down.
-
The distribution node of a logistics site
- The problem
- The distribution node of the site network was unstable and the uplink ran through a faulty optical module.
- What we did
- A new Aruba switch as the MSTP root, the uplink moved off the faulty module, a syslog server and an availability dashboard for 22 devices.
- Result
- The site network is stable and observable; the spanning tree study has been partly applied.
-
An intermittent loop between two aggregation technologies
- The problem
- Some servers were losing traffic intermittently, with no obvious fault, on a backbone of eight devices from two manufacturers spread across three campuses.
- What we did
- We wrote a detector based on a simple principle: the two members of a VLT pair cannot disagree about where a MAC address is. It identified the affected servers, and a study of the configurations located the loop at the point where two different MLAG technologies meet, on a link with spanning tree filtered.
- Where we have got to
- We removed 28 VLANs from one link (CPU on a core switch from 27% to 12%), re-enabled a second link with no moves on 578 MAC addresses, reduced MAC moves from 12 to 2 per minute and brought VLT inconsistencies down to zero with a temporary fix. The structural solution has been studied, with the hardware constraints verified.
-
Router redundancy at a production site
- The problem
- The site’s pair of routers did not guarantee a clean failover: BPDUs were not being exchanged because VLAN 1 was untagged on one side and tagged on the other.
- What we did
- We rebuilt the redundancy with VRRP and an automatic check against loops.
- Result
- Measured failover: about 5 seconds for a router failure, 10 for an unplugged WAN cable, failback with no overlap.
-
Management locked down on 26 routers after a security advisory
- The problem
- A CERT advisory reported RouterOS vulnerabilities being exploited through SSH. We needed to know at once which devices were exposed and to lock them down, without losing access to any of them.
- What we did
- An inventory of 67 devices across 11 projects, with a traffic-light rating based on exposure to known vulnerabilities, not on version numbers. On 26 devices we restricted management access to allowed addresses and switched off unnecessary services, with a mandatory preview, an on-board scheduler that restores everything within 10 minutes if the change is not confirmed, and a check over a new connection.
- Result
- Measured result: no device with SSH exposed. The six-wave upgrade plan has been written and is in progress.
-
Compromise check on a VPN concentrator
- The problem
- Ruling out with certainty that a VPN concentrator had been tampered with.
- What we did
- A complete check of users, keys, scheduled tasks, scripts, tunnels, firewall, routes, DNS, proxy, certificates and change history.
- Result
- No implant found, with an itemised list of what was checked.
-
Roaming fixed across more than a hundred access points
- The problem
- Slow roaming and devices “connected but with no address” at several sites.
- What we did
- We documented that the controller works only at layer 2, while the gateway and DHCP relay sit on the firewall and 802.1X authentication goes to the RADIUS servers. We applied the fixes (802.11r on 20 SSIDs, roaming assistant, DTIM, minimum data rate) with a script and verified them with 57 automatic checks (56 passed), with a targeted rollback on three SSIDs.
- Result
- Roaming has been fixed. For the devices with no address, we also corrected our first hypothesis in writing: what unblocked them was restarting the access points, not disabling 802.11r.
-
Warehouse terminals that changed access point five times an hour
- The problem
- The older warehouse terminals were losing their connection while in use.
- What we did
- From the controller data: on the SSID with 802.11r those terminals made 4.8 roaming events and 3.8 disconnections per hour, against 0 and 0.2 on the SSID without it. There is more than one cause: the threshold for “sticky” clients disabled, an outdated driver, the lack of 802.11k and v, known issues with the FT-PSK combination.
- Where we have got to
- Recommendations delivered in order of priority.
-
The network of a major event, from the bench to the close
- The problem
- A temporary network with many different areas, guests and productions, which had to hold up for the whole event and be manageable remotely.
- What we did
- Gateway and DHCP on a high-availability Linux cluster, expanded from two to four nodes without downtime; two internet lines with policy routing per VLAN; 16 VLANs, some with a bandwidth limit. Seven routers connected to the management VPN: some cloned units had the same bridge MAC and pushed packet loss up to 100%. Eleven access points with the channel plan built around the constraints that apply in Italy (DFS channels, weather radar zones). A web panel for moving a VLAN from one line to the other, tested with 10 trials that brought three defects to light, all of them fixed.
- Result
- The network was in service for the entire event, with redundant monitoring: automatic failback measured at about 18 seconds.
-
Access points prepared on the bench
- The problem
- Preparing new access points in a short time, with the correct firmware, names and addresses.
- What we did
- A bench with the controller on a laptop and the access points connected one at a time over PoE; a PowerShell script adopts them, updates the firmware, renames them, assigns the address and sets the controller address via SSH.
- Result
- About 5 minutes per access point, 8 prepared in one day, with the pitfalls documented for next time.
-
A web application exposed only where needed
- The problem
- An application reachable from the internet was receiving scanner traffic and login attempts.
- What we did
- nginx forwards only two paths and closes everything else without replying; access allowed to a list of sources, a limit of 10 requests per second per address, 21 scanners recognised and rejected, four fail2ban rules fixed because they were not reading the logs.
- Result
- Attack surface reduced to two paths. When an automatic update crashed a third-party module, we identified it from the kernel log and disabled it.
-
Outlook getting the office blocked by the brochure website
- The problem
- The office’s internet address kept ending up on the blacklist of the hosting service for the company website.
- What we did
- We discovered that Outlook was querying the autodiscover service on the brochure website 378 times a day, and the hosting provider’s anti-bot filter mistook it for an attack. A Cloudflare rule, verified against 16 variants, answers those requests before they reach the site.
- Result
- Outlook’s requests no longer reach the site and the office no longer ends up on the blacklist.
-
Video surveillance events that did not reach the monitoring centre
- The problem
- The camera events were not reaching the monitoring centre, yet the links between the servers were all working.
- What we did
- The firewall logs showed that the event channel was travelling over UDP to APIPA addresses, advertised by the software of the servers’ USB management adapters. We disabled those adapters and restarted the service only where needed.
- Result
- Events are arriving regularly again, without touching the rest of the system.
-
PCs that would not update, due to three overlapping causes
- The problem
- Windows updates were failing on many of the office’s PCs.
- What we did
- There were three causes. We resolved two of them on the PCs; using TTL values and response times, we located the third in a transparent proxy run by the operator, after clearing the router and firewall with their counters.
- Where we have got to
- Two causes resolved, the third documented and in the operator’s hands.
-
Network flow collection for a cloud provider
- The problem
- There was no view of who talks to whom and how much.
- What we did
- nprobe as a NetFlow v9 collector feeding ntopng, with the archive on ClickHouse; we defined the local networks.
- Result
- Hosts recognised as local went from 0 to 82, with no flows lost.
-
Data centre and backup Service cluster for several clients (5 nodes in 4 data centres) In production
The memory the file system cache was taking
- The problem
- On one node the ZFS cache was taking up 116 GB of RAM and swap was at 100%; during backups a pool of mechanical disks pushed the load to 62 on 16 threads.
- What we did
- A cache limit on each node and memory deduplication (KSM); a write cap on the disks of the backup repositories.
- Result
- Free RAM went from 12.9 to 68 GB on the most heavily loaded node; during backups the load dropped from 62 to 22.
-
A new cluster sized from measurements
- The problem
- Choosing the second and third nodes of a cluster without buying more than necessary.
- What we did
- We installed Proxmox VE 9 through the provider’s API. We chose the second node based on the real load (0.36 cores on average), single-core performance and the compatibility of the virtual CPUs. Two nodes plus an external vote, ZFS replication every 15 minutes over a private network measured at 918-947 Mbit/s.
- Result
- The third node, installed in 15 minutes, took the place of the external vote.
-
vMotion blocked by a kernel module
- The problem
- At a foreign site with 2 hosts and 31 virtual machines, live migrations were not working.
- What we did
- We removed the ISOs mounted on 7 machines and used cold migrations to reset the corrupted migration state of 26 machines; the root cause was an ESXi kernel module that was not loaded on one of the hosts.
- Result
- vMotion working again in both directions: 14 machines rebalanced live in 11.6 minutes, with no downtime.
-
Two hosts moved into an EVC cluster in four minutes
- The problem
- Reorganising a vCenter with 9 hosts and 97 virtual machines by moving two hosts into a cluster with uniform CPU compatibility.
- What we did
- We carried out a full study of the environment with a critical analysis, then entrusted the move to a tested script.
- Result
- Four minutes in total, machines down for about three, addresses unchanged.
-
A VPN hub moved without interrupting anyone
- The problem
- Decommissioning a virtualisation host at one provider and moving its services to a node at another provider, including the VPN hub used by many users.
- What we did
- We moved the VPN hub (33 OpenVPN clients and 63 WireGuard peers) one client at a time. We adapted the machines restored from backup to the new hypervisor and repointed 35 DNS records across three providers via API.
- Result
- Each client reconnected in about 10 seconds, all DNS zones checked again, the old host verified empty and decommissioned.
-
A database moved with 41 seconds of downtime
- The problem
- Moving the application backend from a virtual machine to a new server without losing data.
- What we did
- The dress rehearsal brought to light a hard-coded path, which we fixed before the switchover. While DNS was propagating, the old machine acted as a reverse proxy to the new one.
- Result
- 41 seconds of downtime, 82,954 records before and after.
-
From one path to four, without stopping the cluster
- The problem
- During maintenance on an array, 12 virtual machines had I/O errors for 23 minutes.
- What we did
- Our read-only analysis found the cause: wrong iSCSI targets in the configuration, a single path per node to the two replicated arrays. We fixed it live, one node at a time.
- Result
- From one to four paths per volume, also verified after a reboot.
-
A Fibre Channel switch replaced without conflicts
- The problem
- Replacing a failed switch in a SAN of four switches across two fabrics.
- What we did
- We brought the new switch up to the version used in the fabric with four upgrades in sequence and transferred the licences from the failed unit.
- Result
- Merged into the fabric with no zoning conflicts.
-
Two storage clusters decommissioned without surprises
- The problem
- Shutting down two NetApp clusters (four controllers) without taking anything away from anyone.
- What we did
- We took a full inventory through the API, then checked that no system was still using them.
- Result
- We decommissioned them only after seeing that nothing was responding any more.
-
A full backup appliance, cleared out without losing copies
- The problem
- The 32 TB deduplication appliance had zero bytes free and the backups of more than 50 jobs were failing.
- What we did
- Before deleting, we checked, machine by machine, where another copy existed. We also discovered that the space shown by the console came from a wrong cache (898 GB against an actual 12,643): since then we decide on the basis of the direct source. Then 24 new jobs from the array snapshots over Fibre Channel.
- Result
- We freed 5.9 TiB on the appliance and more than 14 TB on a ReFS repository; backups running again.
-
A job that failed 211 times in a row
- The problem
- A backup job failed every time, and nobody could find out why.
- What we did
- From the logs of the individual tasks: every task that went through one particular proxy failed (160 out of 160) and all the others succeeded (269 out of 269). We excluded that proxy, with a rollback ready.
- Result
- 9 out of 9 machines backed up in 38 minutes.
-
Night-time crashes and a broken job chain
- The problem
- The backup service went down at night (13 times in 14 days) and a job chain had silently broken, leaving many machines without recent copies.
- What we did
- The causes, one after another: a named SQL instance, a port resolved via UDP, memory under pressure, dropped packets. We stopped the crashes with a static TCP port; we stitched the chain back together and migrated the older jobs in order to move to version 13.
- Result
- Chain restored; 48 jobs migrated to version 13, none failed.
-
Ten terabytes from an unreadable volume, verified byte by byte
- The problem
- A 16 TiB ReFS backup repository had become unreadable; it looked like a disk failure.
- What we did
- The “I/O errors” were the underlying storage running out of space; the disks were healthy. We made a full copy to another node (15.6 TB, verified). On a clone we discovered that Windows Server 2025 upgrades the volume irreversibly even without mounting it: on the original, that would have made the only tool able to read it useless. We carried out the extraction on the clone.
- Result
- 1,479 files and 10.08 TB extracted with byte-by-byte verification in about 3.5 days instead of the estimated 20; Microsoft 365 backup databases repaired, backups running again, residual damage below 1.6%.
-
A production machine recovered with 14 ms of downtime
- The problem
- A production virtual machine was on a corrupted thin pool.
- What we did
- We recovered it from a clone of the array snapshot on an isolated node, then brought it back into the cluster with a live migration.
- Result
- 14 milliseconds of downtime; cleaning up the pool then freed about 23 TiB.
-
The backups found on a disk that had dropped out of the mirror
- The problem
- After an attack, intact copies of the virtual machines were needed.
- What we did
- A disk that had dropped out of the NAS mirror months earlier held an intact backup chain: we mounted it in forced read-only mode and copied it to the new NAS.
- Result
- 826 GB and 4 virtual machines recovered.
-
A disaster recovery site shut down and powered back on entirely remotely
- The problem
- The DR site (virtualised storage, blade servers, 6 hosts and about 145 replica machines) had to be shut down and then powered back on with nobody on site.
- What we did
- A 10-phase shutdown plan with a script that by default only simulates, explicit confirmations for the steps that touch production and safety interlocks. Power-on through an 8-phase orchestrator; we restarted a storage controller stuck at the loader from the server’s remote management console.
- Result
- Verification by actually logging in to 30 devices: 30 out of 30. Then we measured the accumulated RPO gap (about 3 days) and reactivated the 26 replication jobs in a chain.
-
Replication from a foreign site that kept failing
- The problem
- Replication from a foreign site failed on every run.
- What we did
- The cause was a chain of 9 snapshots that had never been consolidated on the target side; we recreated the replicas via API.
- Result
- Replication running regularly again.
-
Restarting after a power outage with the electricity coming and going
- The problem
- A plant had no stable power; the only access left was a 4G/5G management router.
- What we did
- From there a sentinel logged the drops and returns of mains power, with a simple rule: nothing is switched on until the mains has held for 30 minutes. The servers’ management controllers told us which room had power. A 9-phase restart runbook, written using IP addresses because on a cold start there is no DNS.
- Result
- On restart about 90 machines had started up on a single host (RAM at 94%): 61 redistributed live, 61 out of 61 successful. We analysed the database, which had switched over to the secondary node, and prepared the resynchronisation procedure, to be run upon authorisation.
-
A network card that kept disappearing from the bus
- The problem
- The cluster crashed periodically and, after the takeover, one machine in three was responding.
- What we did
- The hardware logs read via Redfish (208 entries, 48 of them critical) always showed the same three PCIe errors: the 10G synchronisation card was disappearing from the bus.
- Result
- Cause proven and the behaviour after the takeover explained too; recommendation delivered.
-
A 132-byte transaction that was filling the disk
- The problem
- The database log was reaching 267-275 GB and filling the disk.
- What we did
- The cause was an external application that left a 132-byte read-only transaction open: even in SIMPLE recovery it was enough to prevent log reuse. An emergency fix and a cap on the log.
- Result
- Disk safe; we pointed out the application fix to its developers.
-
RMAN blocked by archivelogs deleted by hand
- The problem
- RMAN backups would no longer start after some archivelogs had been deleted by hand.
- What we did
- We removed 22,527 orphaned records from the control file, re-registered the catalogue and terminated 11 blocked sessions.
- Result
- Test backup successful; shutdown and restart procedures written.
-
Seven million permissions read in eight minutes
- The problem
- It was necessary to know who could read what on a file server that had grown over the years.
- What we did
- An NTFS permissions scanner written for the purpose.
- Result
- 465,280 folders and 7.3 million permissions analysed in under 8 minutes.
-
The retirement of EWS in Exchange Online, explained to clients
- The problem
- The retirement of Exchange Web Services in Exchange Online risked stopping Microsoft 365 backups.
- What we did
- A guide for clients (tenant settings, propagation times, updates) and the script to apply the settings.
- Result
- A guide and a script that every client can apply on their own.
-
A mail gateway rebuilt alongside the old one
- The problem
- The edge mail server was running on an out-of-support operating system.
- What we did
- We built the new server alongside the old one and tested the SMTP chain before swapping the address.
- Result
- New server in service, with the old one kept ready for rollback.
-
A machine that refused SSH, and it was not the memory
- The problem
- A virtual machine was refusing administrative connections and seemed to be short of memory.
- What we did
- 151 devices were connecting continuously (about 53 connections a minute) and, with the machine slowed down, they saturated the pool of not-yet-authenticated connections; the underlying cause was 16 Java processes with no memory limit. We reduced the logs, measured at 151.9 GiB over 30 days, while keeping the metrics.
- Result
- Cause clarified and load reduced; a network added to the VPN with a runbook and rollback, with 2 minutes and 54 seconds of downtime and verification from three different networks.
-
The heavy midday copy was down to a time zone
- The problem
- Agent-based backups of the Linux machines were failing and a heavy copy was starting in the middle of the day.
- What we did
- Secure Boot was blocking the kernel module used for snapshots: we rebuilt the jobs without snapshots. The midday copy was down to the wrong time zone on the backup servers.
- Result
- Nine machines backed up regularly, at the right times.
-
From 250 notifications a day to zero, keeping the real alarms
- The problem
- A monitoring system for 36 nodes in 8 groups produced permanent alarms and hundreds of flapping notifications.
- What we did
- A two-tier archive (raw data for 7 days, hourly aggregates for 400), thresholds calibrated on historical data, a confirmation delay before notifying. A check on the age of the last backup, read from the database instead of from the job.
- Result
- From 5 alarms, 3 of them permanent, to 2 real ones; flapping notifications down from about 250 a day to zero; a 30-day chart opens in about 25 ms. The backup check found databases without a recent full copy while the job was reported as successful.
-
The monitoring wall for a major event
- The problem
- Keeping track of the devices, lines and Wi-Fi of a temporary network, with a view that can be read from a distance in the control room.
- What we did
- A Python portal with no frameworks or external databases: 20 devices checked every 60 seconds, bandwidth and errors every 10, access point radios and clients every 45; a wall with purely semantic colours, verified with screenshots; alerts on Telegram. We validated the episode-based alarm rule by replaying 24 hours of real data.
- Result
- Notifications down from about 80 to 4 a day. We also found a DNS resolver that took 3-4 seconds per name and access points pushed by weather radars onto reduced-power channels, which can be brought back to 1 W with a button.
-
The new CPU was not needed
- The problem
- Working out whether a node needed upgrading.
- What we did
- Real load measurements: on average 0.41 cores out of 24. We also measured the cost of disk encryption: about 1.5% of a core during peaks.
- Where we have got to
- The limiting factor was the disk, not the CPU: the study shows where it is worth intervening.
-
Exfiltration with extortion: how much had actually got out
- The problem
- The attackers had published part of the stolen data and claimed to have much more. Deciding on the notifications required an objective measure of what had got out.
- What we did
- We acquired the published material on an isolated forensic machine, verified its authenticity with hashes and metadata and analysed it document by document, deriving only aggregate figures: data categories, the presence of special categories of data and of identity documents, validated and masked tax identification numbers. We extracted no personal data in clear text.
- Result
- A factual basis for the notifications, with an explicit boundary between what the sample proves and what remains to be verified in the server logs. In addition, an automatic hourly check of the site where the data was published, with an email alert for every change.
-
Double-extortion ransomware involving special categories of data
- The problem
- Understanding which data the attackers claimed to have, how much of it and of what kind, in order to document the incident for the authorities.
- What we did
- We opened the incident case file (initial collection, investigation plan, structured request for information and evidence, evidence register, timeline) and prepared the tool that classifies the data on the isolated machine, returning only aggregate and masked values.
- Where we have got to
- Case file and tools ready: the analysis continues as the material comes in.
-
Diverted bank transfers: the breached mailbox was not the client's
- The problem
- The company had made several transfers to fraudulent bank details and feared that its own email had been breached.
- What we did
- We analysed the access logs of the internal mailboxes: no anomalous access, no hidden forwarding rules. By cross-checking the message headers we proved that the compromised mailbox was the supplier’s; the counterfeit domain came to light when we computed the distance between the names.
- Result
- Forensic report with a full timeline, indicators and actions: recall of the transfers, blocking of the lookalike domain, verification of the bank details through an independent channel, a mailbox audit and a rule against spoofing of suppliers’ domains.
-
Remote code execution, without a single login
- The problem
- A vulnerable web application, running with elevated privileges, had made it possible to install a malicious program and two persistence mechanisms.
- What we did
- We proved that the attacker had never logged in: the code ran through the application’s process, so the authentication logs were clean by definition. From the application logs we reconstructed which credentials had actually been read and which had only been exposed, then we removed the persistence.
- Result
- Incident scope clarified, stolen credentials replaced and verified one by one, exposure reduced.
-
Notification package for a data processor
- The problem
- The incident concerned data that the provider processed on behalf of its clients: the communications had to take both roles into account.
- What we did
- We prepared the drafts for the Garante, for the data subjects and for the clients acting as controllers, highlighting the fields to be completed, masking the special categories of data and calculating the deadlines: 72 hours for the Garante; 24 hours, 72 hours and one month for the NIS2 stages.
- Result
- Texts ready for review and signature by the DPO and legal counsel, consistent with one another and with the evidence.
-
The SIEM was green, but it was no longer collecting anything
- The problem
- An internal certificate had expired and all the agents had stopped sending data at the same time. The dashboard stayed green and for months nobody had noticed.
- What we did
- We found the cause and reissued the certificate from the same internal authority, so the agents did not have to be re-enrolled one by one. Then we extended the retention of the security log on the domain controllers, installed the missing agents and built a dashboard to track the use of a critical administrative account.
- Result
- SIEM receiving data again, broader coverage and two permanent checks to avoid a repeat of the silent failure: the latest data received from each source and certificate expiry.
-
Virtual machines that looked protected
- The problem
- A large proportion of the machines turned out to be unprotected and some console records gave a false picture.
- What we did
- We cross-checked the real inventory against the console’s, identifying machines by a reliable identifier rather than by name; we installed the missing agents, fixed the cases of antivirus installed but switched off, tuned the policy for a server without a graphical interface and measured the CPU load.
- Result
- Complete and verified coverage, with the remaining anomalies listed one by one.
-
EDR on the servers of a vehicle location system
- The problem
- The Linux servers of the system that locates the fleet’s vehicles had to be brought under EDR coverage, without disrupting the service.
- What we did
- Agent installed via SSH, with registration and full functionality verified in the console; we traced the false positives of the automatic check to names reconstructed from the addresses.
- Result
- Coverage of the server estate verified.
-
Credentials replaced and verified device by device
- The problem
- After a compromise, it emerged that the attacker had read some passwords for network devices and servers.
- What we did
- We changed the credentials of all the devices involved, with a verification round for each one: login with the new password, rejection of the old one, a check that users, scheduled tasks and keys had not been altered.
- Result
- Exposed credentials replaced and verified, and a list of the other systems on which the same passwords had to be changed.
-
From the internet, only port 443
- The problem
- The servers’ administration ports were reachable from the internet.
- What we did
- We left only port 443 open, moved administration behind three independent WireGuard tunnels with multiple paths to each machine, added automatic blocking of failed attempts with an alert on every successful login and wrote the emergency procedures for getting back in.
- Result
- Exposed surface reduced to a single service, with procedures for getting back in even if a tunnel goes down.
-
Cyber Essentials certification
- The problem
- Obtaining the British Cyber Essentials certification, which requires installing high-severity updates within 14 days of their release.
- What we did
- Scheme requirements, questionnaire, device inventory, security bulletin register and update procedures with evidence retention.
- Where we have got to
- Process under way, with the 14-day deadlines kept under control.
-
A hundred and fifty devices responding again within an hour
- The problem
- In a fleet of about 300 supervised iPads, almost all had stopped executing management commands.
- What we did
- Active tests (the command acknowledgement, not the “last seen”), splitting the fleet by enrolment date and logs read over USB: the notification channel certificate had been replaced by one with a different topic. We re-uploaded the original, which was still valid, through an undocumented console step.
- Result
- 149 devices responding again within an hour and at least 1,587 apps installed or updated, without a single physical intervention and without data loss. Then the security posture inventory, with recommendations.
-
A field kit for unresponsive devices
- The problem
- Some devices no longer responded and had to be restored on site, by non-technical staff.
- What we did
- A single executable, with no need for internet access or administrator rights, that backs up the data, erases the device and has it re-enrol automatically.
- Result
- An iPad that had been unresponsive for 28 days, re-enrolled in 23 minutes and responding to commands in 3 seconds.
-
Telephony and contact centres Contact centre in the distribution sector, a few dozen workstations In production
The phone bar a multi-site contact centre was missing
- The problem
- The platform’s standard web agent client was not enough: the microphone stayed blocked inside the page frame and call events were not accessible.
- What we did
- A bar with an isolated web view and deep integration with the platform. Over time it has added automatic forwarding from the physical phone to the virtual extension, the customer record on answer, call pickup from the queue (with counts identical to the native counters), supervisor-initiated pause and switchover to the disaster recovery site.
- Result
- In daily production use, with ongoing development in 2026.
-
A phone bar that started but was not visible
- The problem
- On some workstations the bar showed as running but did not appear on the screen.
- What we did
- The workstation log showed SIP registrations without a single click from the agent. The cause: a VDI image with desktop composition disabled, which does not draw transparent windows.
- Result
- The bar now opens opaque when composition is missing and notes this in the log.
-
Calls assigned to agents already on the phone, reduced by 97%
- The problem
- The automatic call distributor was assigning inbound calls to agents already busy on an outbound call.
- What we did
- By observing the protocol we saw that 88% of the collisions started from the paused state, because of a recovery mechanism in the bar, and that the command signalling “on outbound call” was missing before the call started. Two fixes, then measurement with a server-side observer independent of the bar.
- Result
- Overlaps fell from 23,600 a day, on the peak day, to 608: down 97%.
-
Outbound calls dropping at exactly 90 seconds
- The problem
- Some outbound calls were always cut off after 90 seconds.
- What we did
- We tested and ruled out the session timer, phone and network leads, and demonstrated from the server logs that a timer that had not been reset was putting the agent back into the waiting state while the call was still in progress.
- Result
- Cause fixed and verified.
-
Two registrations per workstation where only one was allowed
- The problem
- Agent registrations were showing as “unreachable” in bursts (2,722 in 14 hours across 64 extensions), with audio interruptions.
- What we did
- We demonstrated, also with the official web client, that each workstation opens two SIP user agents on the same extension, while the configuration accepted only one. A pilot on two extensions, then roll-out to 112.
- Result
- Alerts reduced by 97% in 5-minute monitoring, audio interruptions down to zero, inbound calls verified with a SIP capture: no double ringing.
-
A full disk on the disaster recovery SBC
- The problem
- The SBC at the disaster recovery site had a full disk.
- What we did
- We freed up space by removing 12 million rows from a debug log, without losing a single call record, and extended the disk online.
- Result
- Space recovered and a larger disk, without stopping the service.
-
Two hundred and thirty “suspected dropped calls” a day, none due to the network
- The problem
- An indicator counted about 230 suspected dropped calls a day, 4.9% of the calls handled, and the network was thought to be the cause.
- What we did
- We reconstructed 147 cases one by one: all the repeat calls were new calls and none was a network drop. The causes were mainly three behaviours at the workstation: a microphone with no signal, hold pressed before the call connected, a double click on answer/hang-up.
- Result
- Measurements and safeguards added to the bar, to be enabled after a measurement period.
-
Customers dropped while on hold, or perhaps not
- The problem
- It was estimated that about 200 customers a day were lost while on hold for a consultation.
- What we did
- We counted the real cases from the PBX event log.
- Result
- There were 7 in 14 days, out of 12,076 consultations: the first estimate also counted successful transfers.
-
A trunk that completed fewer calls than the others
- The problem
- Outbound calls were dropping a few seconds after being answered.
- What we did
- The PBX performance counters showed one trunk completing 77% of calls against 89% and 97% for the others; we ruled out the client software, the platform, the SBC and bandwidth control one by one.
- Result
- Cause traced to the telephone carrier and documented in a dossier.
-
A switch to fail over to disaster recovery
- The problem
- In the event of a failure the workstations had to move to the DR site without reconfiguration and without touching DNS.
- What we did
- The bar reads the state from a portal and, if needed, remaps servers, STUN and database to the DR site under the same names, so the certificates remain valid; if the portal does not respond, it stays on production. During testing we found and fixed STUN and dialplan settings still tied to production, which were causing one-way audio.
- Result
- Switchover validated end to end for inbound calls, including the restart of the DR node (about 2 minutes of downtime).
-
Orderly restart of the platform
- The problem
- Restarting the platform’s three servers without surprises for the agents.
- What we did
- A procedure covering database, SBC and web in that order, with 13 final checks.
- Result
- Carried out several times, including before the service opened, always successfully.
-
A PBX for a public-facing support service, compiled from source
- The problem
- A PBX was needed for 16 workstations on an operating system that no longer ships Asterisk.
- What we did
- Asterisk 22 compiled on Debian 13; IVR with an information notice, three round-robin queues with memory, public holidays up to 2035 in the PBX database, ten messages in the same synthetic voice, nftables and fail2ban, SQL triggers for the statistics dashboard already in use, documentation in 12 chapters.
- Result
- PBX delivered and tested with calls generated in the dialplan.
-
The night-time message heard during the day
- The problem
- During the day, callers heard the night-time message.
- What we did
- We compared 31 daily backups of the PBX: 84 modules out of 88 identical, a single relevant change, with its date.
- Result
- Change identified and reverted.
-
Outbound calls blocked by a route with no trunk
- The problem
- On a new PBX, outbound calls were not going out.
- What we did
- The route created by the setup wizard had no trunk assigned: we fixed it via the API. We ran the SIP diagnosis with tcpdump from the hypervisor, because direct access to the PBX was switched off.
- Result
- PBX in service with a queue, opening hours and a night-time message.
-
Physical phone and virtual extension, without thinking about it
- The problem
- The agents had a Cisco phone on their desk and a virtual extension on the platform: calls had to reach the right place.
- What we did
- At login the bar sets forwarding of all calls to the virtual extension via AXL and removes it at logout, with automatic recovery after any abnormal shutdown.
- Result
- Forwarding always consistent with the agent’s state.
-
Main numbers moved within a federation of PBXs
- The problem
- The main numbers of two sites had to reach the PBX at head office.
- What we did
- We changed the inbound routing maps, keeping the old rules as a fallback, saving the configurations before and after and documenting the rollback.
- Result
- Tested and working.
-
Call outcomes, recorded at the end of each call
- The problem
- The end-of-call window, written years earlier, had to be rebuilt.
- What we did
- We rewrote it from C# WinForms to Electron: it shows the contact from the address book (about 8,300 entries), records one of seven outcomes in SQL Server with parameterised queries and updates itself.
- Result
- Tested on the real database.
-
The customer record on answer
- The problem
- The customer record in the business software and in the help desk did not open after a call was picked up from the queue.
- What we did
- We fixed the defect and added the agent code and a filter for internal calls, then checked it live.
- Result
- The record opens on answer and on pickup from the queue.
-
Service level weighted by actual volumes
- The problem
- The daily per-queue dashboards did not give a reliable monthly measure.
- What we did
- We turned 132 daily dashboards, from October 2025 to February 2026, into a monthly report of calls handled and volume-weighted service level.
- Result
- A repeatable monthly report on tens of thousands of calls a month.
-
Authentication pop-ups on every search, eliminated
- The problem
- Every contact search brought up authentication prompts, and one user could not sign in.
- What we did
- A missing attribute in Active Directory was blocking sign-in; the pop-ups came from the server’s OAuth authentication and from photos read from Exchange Online.
- Result
- Sign-in and searches without interruptions; we also documented the decision not to rebuild an integration based on retired technology.
-
External federation unblocked
- The problem
- Some users could not communicate with external organisations in Teams.
- What we did
- We traced the cause to the coexistence mode between Skype for Business and Teams.
- Result
- Two users moved to Teams Only mode, with a documented workaround.
-
The seven ways to handle out-of-hours calls
- The problem
- The service was closed 119 hours out of 168 and out-of-hours calls were lost.
- What we did
- We compared seven options, from transcribed voicemail through an IVR linked to the business software to an AI voice assistant: a latency budget of 800 ms, lines sized with Erlang B, transparency obligations under the AI Act and the GDPR.
- Where we have got to
- Study delivered as a proposal.
-
Two power sources that must never touch
- The problem
- Switching loads between the grid and the photovoltaic (PV) system without the two sources ever running in parallel.
- What we did
- A board with 8 inputs and 8 relays drives two contactors that are also mechanically interlocked, with a 15-minute stability timer, manual operation as the starting state and a return to manual after a loss of PV. Remote access goes through an outbound WebSocket tunnel: we tried WireGuard on the board and rejected it.
- Where we have got to
- In continuous operation since May 2026; a twin board for the heating, stuck in a boot loop caused by a rare chip revision, started up again with a different bootloader.
-
A “USB” port that was an RS-232 serial port
- The problem
- An inverter’s service port, marked as USB, did not respond to any test.
- What we did
- Four failed attempts (USB device, USB host, TTL serial, data cable), then the multimeter: −5 V at idle on the data wire, meaning three-wire RS-232. On the RS-485 bus between inverter and battery, passive listening with Modbus CRC validation showed that the inverter polls 14 addresses with a proprietary function.
- Where we have got to
- Test bench defined and decoder ready, based on the manufacturer’s register map; direct communication is still in progress.
-
A manufacturer's cloud made stable
- The problem
- Pending direct communication, the data came from the manufacturer’s cloud, which kept expiring the session (127 logins in two hours) and renaming fields.
- What we did
- Token-based authentication and tolerant parsing, flagging fields that have disappeared instead of showing a wrong value.
- Where we have got to
- Zero new logins in nine minutes of testing and no battery shown “at 0%” because of a renamed field.
-
Cloud quota used up in two days
- The problem
- The devices’ free cloud quota ran out in two days, at around 26,000 calls a day.
- What we did
- Batched reads with a monthly cap and a service on Raspberry Pi that uses the local protocol, with one open connection per device; the cloud stays as a fallback.
- Where we have got to
- Four devices read and controlled locally, 24 out of 24 acceptance tests passed, six out of six commands from the dashboard.
-
An information wall and the limit of the GPU
- The problem
- A Raspberry Pi 3 had to display a portal at 1920×1080, but the browser was stuck on a version that was too old.
- What we did
- Reinstallation with the current operating system and an up-to-date Chromium; a black screen at boot fixed by changing the video driver; 15 voltage drops in 26 minutes eliminated with a UPS.
- Where we have got to
- A working wall and a written requirement: the Pi 3 GPU supports up to WebGL 1, so interactive maps need a Pi 4 or 5.
-
A DVR without RTSP, republished as a standard stream
- The problem
- A 4-channel analogue DVR offered no RTSP, ONVIF or web interface.
- What we did
- Using the manufacturer’s SDK we found that the proprietary stream contains MPEG-PS with H.264 video: we copied it with ffmpeg, without re-encoding, to an RTSP server.
- Where we have got to
- Four channels available over standard RTSP (704×576, about 25 frames per second).
-
Video latency of up to 27 seconds, brought down to 1.5
- The problem
- Three 4K cameras and a video intercom over VPN arrived with 14-27 seconds of latency, and the intercom’s video arrived broken up.
- What we did
- 10 frames per second at the source, keeping the same resolution, and a direct connection instead of the relay; for the intercom, UDP with a reordering queue.
- Where we have got to
- On the camera we measured, 10 frames out of 10 read with 1.5 seconds of latency; H.264 errors on the intercom down from about 1,600 to 18 every three minutes. The limit then became the upload bandwidth, measured at about 11 Mbit/s.
-
A dark car against the light, from invisible to visible
- The problem
- The smallest model could not see a dark car against the light.
- What we did
- Measurement-based choice of model and per-class thresholds (person 0.50, vehicle 0.35, animal 0.45); inference moved to a dedicated machine.
- Where we have got to
- From 0 to 205 detections on 205 frames; inference on a 4K frame fell from 1,350 to 24 ms.
-
Sirens recognised without neural networks
- The problem
- Recognising a siren in ambient noise without false alarms.
- What we did
- A rule on the spectrum: the tone must rise at least 40 dB above the background of its band for at least 8 seconds. We tested it with synthetic sirens mixed into 12.5 minutes of real audio.
- Where we have got to
- Zero false alarms and the siren recognised in 8 seconds.
-
A rover that kept switching itself off
- The problem
- The factory firmware did not report the battery status (the rover switched off without warning) and left the motors running if the command was interrupted.
- What we did
- New firmware on the ESP32 and a patch to the on-board microcontroller, with the flash memory 98% full: battery reading, shutdown below 6.5 V, a one-second watchdog, gyroscope-guided rotation.
- Where we have got to
- 13 out of 13 acceptance tests passed; a commanded 90° rotation measured at 86.4°.
-
Twenty-five transmitters that did not exist
- The problem
- A wide scan of the spectrum “found” 25 non-existent transmitters.
- What we did
- They were the receiver’s gain curve: comparison with a local moving median and an estimated crystal error, with a stated repeatability of ±3 ppm.
- Where we have got to
- A reliable measurement bench, with its limits written down.
-
An HMI stuck after a blackout, because of a switched-off PC
- The problem
- After a blackout the HMI froze at every start-up; about ten restarts had not helped.
- What we did
- We analysed about 21,000 diagnostic records with a script and documented the exclusion of the other causes: two byte-for-byte identical copies of the project, all PLCs reachable, licences and databases in order. The cause: a PC hosting a server of the distributed system had stayed switched off.
- Result
- Once the PC was switched back on, the HMI started up again. We delivered the incident report, the system fact sheet, the runbook and a checklist for restarts after a blackout.
-
Reports that must stand up to anyone trying to disprove them
- The problem
- A full analysis carried out by several agents produced convincing reports, but not everything they claimed was true.
- What we did
- A second workflow tried to disprove the three main claims of each report by checking files, processes and logs.
- Result
- Some claims did not hold up (automatic backup copies and wake-ups that were declared but did not exist) and were corrected. Since then adversarial verification has been part of the process.
-
The error that never became an alarm
- The problem
- In all ten monitoring sessions, an error in the check itself stayed silent instead of becoming an alarm.
- What we did
- Two separate channels: new items are reported once, active anomalies every time. Then a before-and-after measurement on 21 risks, using the same metric: do they really reach a person?
- Result
- Three hidden faults resurfaced. The measurement showed that notification to people had not yet improved, and since then new critical issues have arrived with an alert from the portal.
-
A quote from eight portals in 40-70 seconds
- The problem
- For a quote we had to open the distributors’ B2B portals one by one, sign in again where sessions had expired and compare prices and availability by hand.
- What we did
- One command queries eight portals, signs in again where sessions have expired and returns net price, list price, availability, promotions and variants, including the actual seller in marketplace offers.
- Result
- Response in 40-70 seconds; automatic sign-in tested on 7 portals. A first version based on a multimodal model that read the pages was replaced by deterministic scripts, which are faster and more predictable.
-
Local tax recalculations, with a check after data entry
- The problem
- Recalculating local tax assessment notices according to the law, case by case, on a web management system without an API.
- What we did
- Seven Python modules, parsers for the notices in two formats, a study of the legislation (penalties, charges, late-payment interest, notifications, the summer suspension of time limits) and eight letter templates. After data entry the program compares the outstanding amount with the letter and, if the difference exceeds the tolerance, cancels the entry and enters it again.
- Result
- In the second batch, 34 of 39 cases completed (2 discarded, 3 passed to manual verification), about 4.6 minutes per case; 22 defects fixed along the way.
-
Supplier invoices downloaded and verified
- The problem
- Collecting the suppliers’ invoices in an orderly and readable archive.
- What we did
- Download through the APIs with the original XML, PDFs generated from the official stylesheet and an index.
- Result
- 16 invoices archived, all checks passed.
-
Reports in the client's corporate format
- The problem
- Producing long reports in the client’s corporate template without redoing the formatting, tables and images by hand.
- What we did
- A generator that clones the original template and preserves its formatting, tables and images, with a suite of automated tests.
- Where we have got to
- It has produced a 43-page progress report with 93 tables and a report on the implementation of NIS2.
-
Company email with no passwords in the scripts
- The problem
- Reading a Microsoft 365 mailbox on a federated domain and preparing replies without entrusting the credentials to a program.
- What we did
- OAuth authorisation with PKCE after a sign-in by the account holder, a refresh token that changes with every use; the replies become drafts in the mailbox.
- Result
- Reading is automatic and sending always stays in the account holder’s hands.
-
Declarations signed and independently verified
- The problem
- Producing digitally signed declarations and being certain that the signature was valid before sending them.
- What we did
- PDFs generated from the official data, remote signing driven by a script, with the PIN and OTP typed in by the holder, then an independent verification: covered bytes, digest, signature over the attributes, certificate.
- Result
- Documents signed in PAdES and the case closed.
-
Five hundred contacts in two mailboxes, zero errors
- The problem
- Importing an address book into two Exchange mailboxes.
- What we did
- Import through Exchange Web Services, with a check of the results.
- Result
- 504 contacts imported, zero errors.
-
A monitoring portal built and improved every day
- The problem
- Bringing together in one place the status of many monitored perimeters, with separate documents for each client.
- What we did
- A portal with per-client views, a page for each issue, PDFs for individual clients verified by an automated test on the text and alerts only for new critical issues.
- Result
- In production since 17 October 2026, with continuous improvements.
-
Do not disturb during video calls
- The problem
- Incoming calls interrupted users’ video calls.
- What we did
- A Windows application recognises video calls from the audio level and the window titles and sets the “do not disturb” status; deployment through 64-bit MSI packages with silent installation.
- Result
- Version 2.4 delivered; the rewrite in Rust and Tauri is a prototype.
-
Navigation on the slopes, even without a network
- The problem
- Finding your way on the ski slopes even where the phone has no coverage.
- What we did
- An Android app with Kalman-filtered GPS, a calibrated compass, route calculation with Dijkstra’s algorithm on a graph derived from OpenStreetMap, elevations and data available offline.
- Where we have got to
- Working prototype, distributed only for internal testing.
-
A coding assistant and the GDPR
- The problem
- Finding out whether an AI coding assistant keeps data in Europe.
- What we did
- Analysis of the provider’s terms: on the standard service, data is also processed outside the European Union; it stays in the region only with the data-residency enterprise edition and a dedicated rule, which is disabled by default.
- Where we have got to
- A documented overview, including what leaves the region in any case: telemetry, billing, support and checks for secrets.
-
Each client sees only their own data
- The problem
- A portal that serves multiple clients must never mix their data in the documents.
- What we did
- Separate PDFs for each client with an automated test on the text, scanning for secrets in the reported issues (only hashes reach the server), pages without external resources.
- Result
- Every PDF passes an automated test on the text before it is delivered.