Coffee Space 
NOTE: This article is currently a work in progress. It may not have an interesting resolution, but you get to see the process in action.
I used to use Thunderbird for my RSS feeds, but a while back it entirely corrupted my feed and I had to put in a lot of effort to recover my subscriptions. I managed to get back to an OPML file and simply decided in the end to move to a different feed reader.
After looking around, I found akregator seeming to be a simple, old and well supported feed reader. From what I can tell it compiles down to a binary, rather than being one of these interpreted language programs that one day decides it doesn’t want to work, because a Node package that does some simple task is suddenly deleted.
Akregator has some problems, like not being able to switch off the tabs system (I only want to download content or view it in a browser), supporting Javascript (my feed reader shouldn’t have a Javascript engine, that’s what my browser is for) and when it crashes it forgets what you have already read (which seems pretty dumb).
That said, it generally has been more reliably than Thunderbird so far, which probably is a pretty low bar to pass - given it corrupted my feeds and silently stopped updating them.
The other day my laptop running Ubuntu started freezing up and completely locked up. All software updates are disabled (including snap), which was the previous cause of instability. After running for weeks, I was surprised that it suddenly became so unstable.
I of course was forced to reboot my machine manually and was quite cautious thereafter, checking back at the System Monitor regularly. To my surprise. I see akgregator of all things consuming over 1GB of RAM! It was giving Firefox a run for its money, and my version of Firefox has over 1000 tabs open! What the hell!
First things first, I went to DuckDuckGo and searched for similar bugs - to which I found none. This was kind of annoying, so I sat on the problem a little longer, played with the settings and the RAM usage continued to increase.
I then went to IRC on the #kde-bugs channel on the Libera server. There somebody recommends to lodge a bug report over on KDE. I check for existing bugs there to do a cursory glance to try and prevent replication of efforts. After scrolling through I find nothing.
After finding the relevant version information, I registered and submitted a report. Some details from that report:
0001 SUMMARY 0002 0003 I am currently observing ~250MB of RAM usage increase per day. This lead to a 0004 system crash. 0005 0006 Currently Akregator is consuming 1.7GB of RAM, 4.9GB of virtual memory. 0007 Compare this to Firefox with hundreds of active tabs, consuming 1.4GB of RAM 0008 and 8.9GB of virtual memory. 0009 0010 STEPS TO REPRODUCE 0011 1. Large number of RSS feeds (OPML export is ~300kB and ~700 lines) 0012 2. Set checking time to 30 minute intervals 0013 3. Wait for 24 hours 0014 0015 OBSERVED RESULT 0016 0017 A ~250MB RAM increase over 24 hours. Over several weeks this will lead to a 0018 system lock-up/crash (in this case). 0019 0020 EXPECTED RESULT 0021 0022 A little to no RAM increase over 24 hours. 0023 0024 SOFTWARE/OS VERSIONS 0025 Windows: - 0026 macOS: - 0027 Linux/KDE Plasma: Ubuntu 21.04 0028 (available in About System) 0029 KDE Plasma Version: - 0030 KDE Frameworks Version: 5.80.0 0031 Qt Version: 5.15.2 0032 0033 ADDITIONAL INFORMATION
Then I thought to myself, perhaps there is more I can do.
After looking online at several solution for monitoring RAM usage for a given process, I settled on and adjusted an answer from StackOverflow:
0034 delay=300
0035 pid="$1"
0036
0037 top -b -d $delay -p $pid | awk -v OFS="," '$1=="top"{ time=$3 }
0038 $1+0>0 { print time,$1,$NF,$9,$10; fflush() }'
After running for about 9 hours, we see this:
As you can see, RAM usage is increasing over time as I suspected. It appears to be increasing approximatly 20MB per hour and is mostly linear. There appear to be two lines, perhaps the top one is when it is actively processing new incoming RSS feeds.
Other than this, it’s still too early to draw conclusions. I now continue to wait for 24 hours and see if somebody becomes interested in my ticket.
As I suspected, things didn’t get better inside the 24 hours observation time:
Next steps:
[x] Upload the data to the KDE bug ticket and explain my next debugging steps to them.[x] Setup valgrind and see if I can’t gleam some useful information about wheter somethign is actually being leaked, or this is simple a design problem (i.e. try to hold all resources in RAM).I have begun running valgrind using the advice from StackOverflow:
0039 valgrind --leak-check=full -v --track-origins=yes --log-file=log.txt valgrind
I did find that --track-origins=yes was exceptionally slow, so I removed this to get something like a usable experience. I will run for a few hours before closing to ensure I definitely get some data.
After testing - is there a leak? Yes. From the log we see:
0040 ==595152== LEAK SUMMARY: 0041 ==595152== definitely lost: 298,872 bytes in 357 blocks 0042 ==595152== indirectly lost: 2,469,366 bytes in 10,308 blocks 0043 ==595152== possibly lost: 232,592 bytes in 476 blocks 0044 ==595152== still reachable: 10,728,421 bytes in 104,272 blocks 0045 ==595152== of which reachable via heuristic: 0046 ==595152== newarray : 936 bytes in 5 blocks 0047 ==595152== multipleinheritance: 29,552 bytes in 21 blocks 0048 ==595152== suppressed: 0 bytes in 0 blocks 0049 ==595152== Reachable blocks (those to which a pointer was found) are not shown. 0050 ==595152== To see them, rerun with: --leak-check=full --show-leak-kinds=all 0051 ==595152== 0052 ==595152== Use --track-origins=yes to see where uninitialised values come from 0053 ==595152== ERROR SUMMARY: 14687 errors from 1145 contexts (suppressed: 0 from 0)
From what I can tell (with my very limited knowledge), we see to be leaking memory mostly from libQt and more specifically libQt5WebEngineCore:
0054 ==595152== 836,845 (186,208 direct, 650,637 indirect) bytes in 92 blocks are definitely lost in loss record 25,600 of 25,601 0055 ==595152== at 0x4842FB3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) 0056 ==595152== by 0x744FCC9: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0057 ==595152== by 0x77B2AB2: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0058 ==595152== by 0x9E17A8A: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0059 ==595152== by 0x9E184AE: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0060 ==595152== by 0x9E1C912: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0061 ==595152== by 0x9E1F664: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0062 ==595152== by 0x9E14D44: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0063 ==595152== by 0x9E1505D: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0064 ==595152== by 0x9E32CE0: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0065 ==595152== by 0x97700DA: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3) 0066 ==595152== by 0x9783A93: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
To be sure, I would probably need to compile from source. If somebody does eventually answer the ticket, they will likely ask me to use the most up-to-date version and confirm the problem still persists there too. For now though I will wait for somebody to reply and keep logging RAM increase over time.
After running for a while longer, the trend appears to still hold true, even from a recently started process:
I thought some more about building from source, and after a little searching, found Akregator on GitHub. I saw there a comment about a dedicated channel on IRC, but haven’t been able to get hold of anybody there yet:
0067 IRC: irc://irc.libera.chat/#akregator
I asked about build dependencies on IRC, but got no comment there. I decided I would just give it a go and see what happens.
First of all, we need to clone and try an initial build:
0068 cd $WORKING DIRECTORY # Got to where we want to clone it 0069 git clone git@github.com:KDE/akregator.git 0070 cd akregator/ # Go to where we clone it 0071 gitk --all # Check everything looks good 0072 git checkout v20.12.3 # Checkout the version I want to build 0073 cmake . # Attempt to generate the Makefile
It gives it a go, but fails:
0074 -- The C compiler identification is GNU 10.3.0 0075 -- The CXX compiler identification is GNU 10.3.0 0076 -- Detecting C compiler ABI info 0077 -- Detecting C compiler ABI info - done 0078 -- Check for working C compiler: /usr/bin/cc - skipped 0079 -- Detecting C compile features 0080 -- Detecting C compile features - done 0081 -- Detecting CXX compiler ABI info 0082 -- Detecting CXX compiler ABI info - done 0083 -- Check for working CXX compiler: /usr/bin/c++ - skipped 0084 -- Detecting CXX compile features 0085 -- Detecting CXX compile features - done 0086 CMake Error at CMakeLists.txt:8 (find_package): 0087 Could not find a package configuration file provided by "ECM" (requested 0088 version 5.75.0) with any of the following names: 0089 0090 ECMConfig.cmake 0091 ecm-config.cmake 0092 0093 Add the installation prefix of "ECM" to CMAKE_PREFIX_PATH or set "ECM_DIR" 0094 to a directory containing one of the above files. If "ECM" provides a 0095 separate development package or SDK, be sure it has been installed. 0096 0097 0098 -- Configuring incomplete, errors occurred! 0099 See also "./CMakeFiles/CMakeOutput.log".
So here we can see it failed to build based on ECM, so we need to find out what it is complaining about and likely install something to meet that dependency. This is the list of things I ended up needed to download via sudo apt-get install <PACKAGE> (your milleage may vary):
extra-cmake-modules (This one appears to just be a CMake extension)libgrantlee5-devlibkf5crash-devlibkf5doctools-devlibkf5kcmutils-devlibkf5notifyconfig-devlibkf5parts-devlibkf5texteditor-devlibkf5notifications-devlibkf5syndication-devlibkf5grantleetheme-devlibkf5kontactinterface-devlibkf5libkdepim-devlibkf5libkleo-devlibkf5messageviewer-devlibkf5pimcommon-devlibkf5messagecore-devlibkf5mimetreeparser-devlibkf5webengineviewer-devlibkf5akonadimime-devFinally I was able to run cmake . and get a good message (the list below of REQUIRED packages may give you a better list for installation):
0100 Installing in /usr/local. Run ./akregator/prefix.sh to set the environment for akregator. 0101 -- Could NOT find KUserFeedback (missing: KUserFeedback_DIR) 0102 -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY 0103 -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success 0104 -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY 0105 -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success 0106 -- Performing Test COMPILER_HAS_DEPRECATED_ATTR 0107 -- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success 0108 -- The following OPTIONAL packages have been found: 0109 0110 * QGpgme 0111 0112 -- The following REQUIRED packages have been found: 0113 0114 * ECM (required version >= 5.75.0) 0115 * Qt5Qml (required version >= 5.15.2) 0116 * Qt5QmlModels (required version >= 5.15.2) 0117 * Qt5Quick (required version >= 5.15.2) 0118 * Qt5WebChannel (required version >= 5.15.2) 0119 * Qt5Positioning (required version >= 5.15.2) 0120 * Qt5WebEngineCore (required version >= 5.15.2) 0121 * Qt5WebEngineWidgets 0122 * Qt5PrintSupport 0123 * Qt5 (required version >= 5.13.0) 0124 * Grantlee5 (required version >= 5.2) 0125 * KF5Crash (required version >= 5.75.0) 0126 * KF5DocTools (required version >= 5.75.0) 0127 * KF5KCMUtils (required version >= 5.75.0) 0128 * KF5NotifyConfig (required version >= 5.75.0) 0129 * KF5SyntaxHighlighting (required version >= 5.80.0) 0130 * KF5TextEditor (required version >= 5.75.0) 0131 * KF5Notifications (required version >= 5.75.0) 0132 * KF5Syndication (required version >= 5.75.0) 0133 * KF5Parts (required version >= 5.75.0) 0134 * KF5KontactInterface (required version >= 5.16.3) 0135 * KF5Libkdepim (required version >= 5.16.3) 0136 * KF5MessageCore (required version >= 5.16.3) 0137 * KF5MimeTreeParser (required version >= 5.16.3) 0138 * KF5MessageViewer (required version >= 5.16.3) 0139 * KF5Sonnet (required version >= 5.80.0) 0140 * KF5PimTextEdit (required version >= 5.16.3) 0141 * Qt5WebEngine (required version >= 5.13.0) 0142 * KF5WebEngineViewer (required version >= 5.16.3) 0143 * KF5AkonadiMime 0144 * KF5ItemModels (required version >= 5.75.0) 0145 * Qt5Gui (required version >= 5.13.0) 0146 * KF5Service (required version >= 5.80.0) 0147 * KF5Completion (required version >= 5.80.0) 0148 * KF5ItemViews (required version >= 5.80.0) 0149 * KF5JobWidgets (required version >= 5.80.0) 0150 * KF5Solid (required version >= 5.80.0) 0151 * KF5Auth (required version >= 5.80.0) 0152 * KF5WidgetsAddons (required version >= 5.80.0) 0153 * KF5ConfigWidgets (required version >= 5.80.0) 0154 * KF5XmlGui (required version >= 5.80.0) 0155 * KF5WindowSystem (required version >= 5.80.0) 0156 * Qt5Network (required version >= 5.14.0) 0157 * Qt5Test (required version >= 5.13.0) 0158 * Gettext 0159 * KF5Contacts (required version >= 5.75.0) 0160 * KF5GrantleeTheme (required version >= 5.16.3) 0161 * Qt5Widgets (required version >= 5.13.0) 0162 * KF5AkonadiContact (required version >= 5.16.3) 0163 * KF5CoreAddons (required version >= 5.75.0) 0164 * Qt5Core (required version >= 5.14.0) 0165 * KF5Codecs (required version >= 5.75.0) 0166 * KF5Mime (required version >= 5.16.3) 0167 * KF5IMAP (required version >= 5.16.3) 0168 * KF5PimCommonAkonadi (required version >= 5.16.3) 0169 0170 -- The following OPTIONAL packages have not been found: 0171 0172 * KUserFeedback (required version >= 1.0.0), User Feedback lib 0173 Allow to send Telemetry Information (optional). It can be disable in apps. 0174 0175 -- Configuring done 0176 -- Generating done 0177 -- Build files have been written to: ./akregator
And now we can see the built binaries:
0178 $ ls bin/ 0179 akregator akregatorstorageexporter 0180 akregator_config_advanced.so downloadfeediconjobtest 0181 akregator_config_appearance.so kontact_akregatorplugin.so 0182 akregator_config_archive.so libakregatorinterfaces.so 0183 akregator_config_browser.so libakregatorinterfaces.so.5 0184 akregator_config_general.so libakregatorinterfaces.so.5.16.3 0185 akregator_config_plugins.so libakregatorprivate.so 0186 akregator_mk4storage_plugin.so libakregatorprivate.so.5 0187 akregatorpart.so libakregatorprivate.so.5.16.3
Next we run the program and make sure the memory leak still exists in the version we just built from source:
0188 ./bin/akregator
This will take a little while, so I will follow up in a further update. The only other thing to do was update the ticket regarding my current actions and plans.
After nearly 24 hours, we observe our RAM study on the built-from-source version:
Some interesting observations:
Settings > Configure Akregator causes a crash:0189 ./bin/akregator: symbol lookup error: /usr/lib/x86_64-linux-gnu/qt5/plugins/akregator_config_userfeedback.so: undefined symbol: _ZN9Akregator19UserFeedBackManager4selfEv
The built version suggests the following versions (supposedly the same as before):
5.16.3 (20.12.3)5.80.05.15.2 (built against 5.12.2)Not entirely sure where the different is coming from here… Time to update the ticket!
I recieved a comment on the ticket that suggests I should remove the local copy of Akregator first:
You rebuild akregator but it seems that it uses akregator_config_userfeedback.so from your system not you install directory. If you rebuild remove your akregator from your system first.
I run sudo apt-get remove akregator and sure enough, it removes userfeedback:
0190 Reading package lists... Done 0191 Building dependency tree... Done 0192 Reading state information... Done 0193 The following packages were automatically installed and are no longer required: 0194 kuserfeedback-doc libkf5messageviewer-plugins libkuserfeedbackcore1 0195 libkuserfeedbackwidgets1 qml-module-org-kde-userfeedback 0196 Use 'sudo apt autoremove' to remove them. 0197 The following packages will be REMOVED 0198 akregator 0199 0 to upgrade, 0 to newly install, 1 to remove and 25 not to upgrade. 0200 After this operation, 6,495 kB disk space will be freed.
After running cmake ., I could see I need to install kuserfeedback-dev with:
0201 sudo apt-get install kuserfeedback-dev
After this I rebuild Akregator to ensure it’s built against local sources:
0202 make clean 0203 make 0204 sudo make install # Install metakit
Now I am re-running the RAM study to see if the issue replicates. Lastly, I update the ticket and we see how things go.
After running Akregator for about 24 hours, we finally see we are able to replicate the issue when building from source (and in debug mode):
Next we want to run Akreagtor under valgrind so that we can get more useful debug information:
0205 valgrind --leak-check=full -v --track-origins=yes --log-file=akregator-vg-2021-08-09.txt ./bin/akregator
Again, running with --track-origins=yes is insanely slow, to the point where I’m not sure the program is usbale and able to replicate the original issue.
Lastly, we update the ticket.
The next steps will be:
valgrind v20.12.3 debug build - We hope to see more useful output in Valgrind using the debug symbols, whoich should hopefully point to exactly where memory is being leaked (if any).The ticket has been updated with the valgrind output.
The next few days will likely be busy, hence I will need to move onto other projects. I will give the person responding on the ticket a chance to check things over and make recommendations - as I’m essentially just stabbing in the dark at this point.
No further updates at this time - except I may have a branch of Akregator to checkout with a different archive back-end that should reduce RAM usage. Apparently, the default behaviour of Akregator is to try and store everything in RAM in order to keep some speed in the GUI.
Given how fast disks are (even old IDE drives), they should be more than fast enough to fetch some feed titles in a fast (undetectable) time. A design for any program that means it slowly uses more and more RAM over time - seems awful and broken. I’m beginning to believe that I might be better off rolling my own feed agreagtor. I don’t have the time for that right now, but I really suspect it really is the right way forwards.
Again, no further updates to report. Still not heard anything else from the developer. As time goes on, the idea of rolling my own feed agregator becomes more attractive. I suspect that the software would require a large overhaul as this is a design flaw - currently it tries to store everything in RAM.
I’m going to have to neglect this from my mind as I have more important things to work on. If I come back to it randomly in the future and there is still no update, I will consider it a lost cause.
Stay tuned for more!