Hardware

After realising there was an issue with v1 as highlighted in the previous article, I made a v2 version. Shortly after submitting this board, I discovered there was still an issue, hence a v2b version.

At least I have some PCBs to give away…

I finally got v2b in the post, buzzed all the connections successfully, and decided to solder it. To save some money I didn’t get a stencil, so I hand-placed solder paste by eye. I then manually pick-and-placed the surface mount components, and to my surprise - it all went well!

Once I was able to confirm that the battery charging circuit could properly charge the battery (by checking the ADC currently sees 4.2V), I was then able to solder the battery.

As you can see, there is very little mass to the battery and it did not want to sit on the pads during soldering. I resulted to taping the LiPo to a plastic case (that the OLED display was shipped in). You obviously can’t tape the battery to the board, because it gets too hot.

It was at this point, I greatly regretted having such a large ground plane. My little USB soldering iron could hardly get hot enough to begin to melt the solder, and as a result the LiPo battery got far hotter than I was happy with.

During soldering, the watch of course sprung to life, indicating that it had been done successfully.

Core Interface

The core interface has come along nicely, with several graphics utilities developed to support it. One is the ability to dump the screen buffer so that I can produce “screenshots” from the device. These are stored as P4 PBM binary format - an old but simple format.

The main clock interface currently shows the measured battery percentage (4.2V 100% - 3.2V 0%) - and this works really quite well given that it is just an ADC reading via a switched ladder.

The time itself is grabbed via the RTC (which survives light and deep sleep), and is currently pushed via a configuration file when the Python files are deployed. It appears to keep time pretty well.

Micropython currently only supports 8x8 pixel fonts, which raises the question: How on earth is there a 16x16 font? We draw the font to the frame buffer, then perform an integer scaling to the frame buffer. In theory I could attempt some form of anti-aliasing, but instead we just embrace the blocky-ness.

Several icons have been prototyped as app icons and icons within apps. The artificial limitation of 8x8 icons is quite challenging. It is technically possible to do vector graphics, but it’s a lot of work. The icons again are P4 PBM binary format, as output by mtPaint.

The menu itself is pretty slick, with the icon directory searched for a same-name icon to display (scaled up), or the first letter of the app name being used instead. In the middle of the display is the full name of the app highlighted to be clear which app is highlighted.

The border around icon is actually a vector graphic and would technically scale to any resolution display. The icon is pre-computed on menu app start to prevent too much processing power being wasted - a common theme in this watch.

Calculator

One of the first apps to be developed was the calculator, because it is simple and shows off the use of Python very well.

Normally the implementation of a calculator app that can handle brackets, etc, is quite a pain. But with Python the values can be easily evaluated. Very long expressions can be expressed and resolved quickly - more than good enough for such a lightweight watch. Dare I say it - this calculator is better than most default calculators in mobile phones.

Beyond the expected functionality, C clears the display and y writes the previous answer. You could for example do the following:

67 + 1 = # 68
y + 1 = # 69

Being able to leverage the simplicity of Python allowed this app to be developed in mere hours.

Power

One of the features I wanted to add was a WDT, i.e. a watch-dog timer. The idea is that if you don’t keep responding to the timer, it resets the device. In this way, it becomes impossible for an app to enter into an infinite loop and crash the entire watch.

It turns out though that once started, you are not allowed to stop the WDT in Micropython, despite it being supported by most modern micro-controllers. The issue really appears when you enter a light/deep sleep for a longer period than the WDT, and upon resuming the CPU the WDT is instantly triggered. This pretty much means that the WDT cannot be used in low-power applications. There is a GitHub issue to hopefully resolve this one day.

Power consumption has been a serious concern, and until now, no effort had been made to start the WiFi. After doing so, the device immediately browned out (crashed due to low CPU power). Uh oh. The problem is that I used a super cheap 300mA 3.3V low voltage drop-out regulator, and during WiFi initialisation, it exceeds this current draw. The reason for using such a low value is that it was cheap, at just 8p each.

I therefore had to turn the WiFi on carefully. To give you an idea:

# active()
#
# Turn the interface on/off carefully to reserve power usage.
#
# @param self Reference to self.
# @param act True to turn on WiFi, otherwise False.
def active(self, act = True) :
  if not self.wifi.active() == act :
    self.wifi.active(act)
  elif act :
    # Test that the interface is *really* working
    try :
      rssi = self.wifi.status("rssi")
    except :
      # Trigger reset of interface
      self.wifi.active(False)
      self.wifi.active(True)
  to = time.ticks_ms() + 500
  while time.ticks_ms() < to and not self.wifi.active() == act :
    time.sleep_ms(100)
  if act :
    self.wifi.config(
      txpower = 10, # 10 dBm
      pm = network.WLAN.PM_POWERSAVE,
    )
  return

You first have to turn on WiFi to set the power configuration options, otherwise that part of the controller is simply not away to listen to the configuration requests. Sometimes it even pretends to be on, but it’s actually not on, so you have to make sure that it’s not tricking you. Even then, you still have to wait for the WiFi to actually become alive and ready to talk.

The real tricks are to put it into a power saving mode (PM_POWERSAVE) which turns down beacons, etc. You must also turn down the transmission power from approximately 20dBm to 10dBm, otherwise it consumes too much power during transmission.

If you do all of these things, then you can get the WiFi to stay on reliably and not brown out the entire CPU.

Future Apps

Now that WiFi is working, the really cool stuff can happen. I have written a http_get() function that opens a socket, grabs the data within a limited buffer and closes the socket, to any given URL at any given port.

The intention is to build out a few features:

• Grab the time from an NTP server
• Read weather predictions via a free API
• Read RSS feeds

In the future, I would then be interested in connecting to a server that can sync information across devices (i.e. calendar, notes, etc), or prepare media such as pictures, videos, web searches, wikipedia entries, etc.

For the most part, I want as many features as possible to operate correctly without needing the internet. If it’s possible to be done without the internet, it should be done so.

Some out of the box ideas that I’ve not seen elsewhere:

• Offline translation for basic things. For example, you could select the Japanese language, select that you want to find a facility, and select between items such as a shop, toilet, etc. It then displays the Japanese characters and you show this to the Japanese person.
• Currency conversion. When abroad it can be difficult to figure out how much things cost in a currency you can understand. The latest conversion rates can be cached or manually entered. “This thing for ten thousand yen, is it a good deal or is it going to wipe out the holiday budget?”

Future Hardware

If I do another spin of the watch, I would consider a few more features:

• Buttons need to be side-mounted
• A method to detect that the device is charging
• Secure display and mount closer
• Remove massive ground plane
• Simplify reset & boot buttons (maybe a single button or temporary bridge)
• Smaller download header (pins 2, 3, and 6 currently not connected)

Some nice to haves:

• IMU - would be nice for waking with arm up gesture, even better with a magnetic compass
• GPS - no idea how this would be powered or fitted
• Microphone - allow the user to interact with the device via audio
• Speaker - play sounds to the user, uses a lot of power
• Vibrator - notify the user, uses a lot of power

Problem

The overall problem that we are trying to solve is the following:

A small productivity watch that is internet connected, to perform a series of lightweight tasks that are synced with other devices via the internet.

Examples of tasks of interest:

• Display the time and date, manually override these and set/display different time zones.
• Connect to a WiFi router with an SSID/password.
• Synced capabilities:
  • Set multiple alerts throughout the day, that are synced to other devices.
  • Check a series of RSS feeds, which can also be synced. I.e. to check the latest news.
  • Check synced notes.
  • Check tasks and set as completed.
  • Check a simple calendar.
• Local capabilities:
  • Unit conversions.
  • Currency conversions (with latest conversions from online sources).
  • Calculator.
  • Meeting calculator with multiple time zones.
  • QR code/barcode/image viewer (useful for scanning into events for example).

Maybe I can think of other things, but I think these are already useful services!

I have used some other devices…

Proposal

Now for the chosen parts…

Parts

The main controller chosen is the ESP32-S3-WROOM-1. It runs micropython quite well from previous experience.

The battery is 30mAh 401012 (dimensions: 4mm x 10mm x 12mm). Assuming a total on time of 30 seconds per hour at 300mAh, and assuming the system is in a light sleep or lower with ~1mAh draw, we can expect the system to run on a full charge for:

batt_mAh = 30
on_mAh = 260 # WiFi average peaks
off_mAh = 1
on_ratio = 0.5 / 60 # minutes per hour
usage_mAh = (on_ratio * on_mAh) + ((1 - on_ratio) * off_mAh)

hours = batt_mAh / usage_mAh
print("Expected usage of " + str(hours) + " hours")

Then we are just touching on a 10 hour operating time.

The ADC IC is the MCP3021A7T-E/OT which takes a supply between 2.7V and 5.5V, and can read within the same range. Foreshadowing: It is not not be.

The voltage regulator is the TLV74033PDBVR, chosen because it is just 8 pence. It’ll take an input between 1.4V and 5.5V and produce a 3.3V voltage. It will output 300mA, which exactly matches the expected spike current from the ESP32 during WiFi. There is no absolute maximum current specified, so I assume we can maybe push the envelope slightly.

Otherwise the next highest current eating device is the 0.96” OLED display with the SSD1306 driver, requiring ~20mA of power at maximum contrast. Everything else is sipping power.

The chosen battery charging circuit is the MCP73812T-420I/OT, which should charge a single cell LiPo at 50mA when the USB supply is provided. It has constant voltage, constant current, thermal regulation, etc.

Implementation

The first thing to notice is that it is a two-layer board and everything is mounted single-side. This means that it can be manufactured simply on a hot plate. The width and depth ~4cm x ~4cm, and the height will likely be defined by the top of the OLED to the bottom of the soldered through-hole headers. Offering the OLED to an existing board shows a height ~17mm. The battery should fit nicely under the display.

As you can see, I have chosen a ESP32 module and not the chip, so it is certified and could be sold. The ESP32 module does take up the majority of the design, though.

The only traces that were manually routed were the USB differential pair, and if you look closely, there is a length matching wiggle near the USB socket. It should be close enough for basic USB 2 communications.

As you can see in the 3D render, there are clear cut-outs for a watch strap. I look to further reinforce this with the 3D printed chassis. The chassis is expected to attach as a “press fit” to secure it in place.

Soldering will need to start with the surface mount components, then the through-hole components, then finally the LiPo. The LiPo will be a pain, but time will need to be taken to ensure it is not directly touched by heat.

Manufacturing

The PCB is sent for manufacturing, and I got quite a good deal on it. Five PCBs for $2, $1.50 shipping charge from China, $0.70 tax, for a total of $4.20. It’ll take a little while to arrive, but everything does.

The real downside is that I didn’t get a stencil. I’m thinking that I could probably cut one out on the vinyl cutter, or make a cardboard single-use stencil.

LLM Review

I was surfing the World Wide Web as I sometimes do, and I came across an LLM netlist checking tool. I plugged in the datasheets for the ICs I use, and gave the following prompt:

“It is a small ESP32 based smart watch with two input buttons, and an OLED display via I2C. There is a battery charge circuit for a single cell LiPo, a voltage I2C ADC to monitor battery voltage, and a 3V3 voltage regulator for the ICs.”

I wasn’t expecting much, I had already sent the board off to be manufactured. But after a few minutes, it generated this:

Overview

Key blockers before layout: the Li-ion charger is effectively unpowered, the 3.3 V rail lacks both the correct regulator variant and the minimum capacitors, and the battery monitor will overdrive its ADC input. Details by subsystem follow.

1. Power-input & Li-ion charging (U2 MCP73812, J1 USB-C, J4 battery)

• Charger supply/enable not wired: Pins VIN (pin 4) and CE (pin 1) of the MCP73812 share net N$11, but N$11 never reaches the USB 5 V rail, so the IC never powers up and CE never sees the required logic high. Tie both pins to the 5 V source and keep CE accessible if you plan to gate charging.[^mcp-pin]
• Mandatory capacitors missing: The charger needs ≥1 µF from VDD to GND and ≥1 µF from VBAT to GND for stability.[^mcp-cap] Add them close to the IC.
• Charge current check: R7 = 2 kΩ programs ≈500 mA fast-charge current.[^mcp-rprog] Confirm the USB source, battery, and PCB traces can safely handle that.
• Battery connector: Nothing obviously wrong; just ensure pack protection/NTC (external) is addressed elsewhere.

2. 3.3 V regulation (U1 TLV7xx, 3v3 net)

• MPN/value mismatch: Net 3v3 relies on U1, but the value is TLV70012 (1.2 V output). Verify the intended part; if you need 3.3 V pick TLV70033/TLV74033 etc.
• Input/output caps missing: TLV74x devices require at least one 1 µF ceramic on VIN and one on VOUT located close to the pins for stability and transient response.[^tlv-cap] Add them.
• VIN source: Once U2 is fixed, route USB 5 V (or battery when USB absent) cleanly into VIN with appropriate filtering.

3. MCU & digital core (J3 ESP32-S3-WROOM-1, SW1–SW4, R3–R6, C3–C4)

• Decoupling absent: The module needs the recommended 22 µF bulk plus 0.1 µF high-frequency capacitor right at its 3.3 V pins.[^esp-supply] Add them on the 3v3 net.
• Boot/strap networks: RC networks (R3/C3, R5/C4) and buttons appear to target strapping pins. Double-check the chosen resistor values meet Espressif’s strap logic levels and that any buttons pulling to 3.3 V have corresponding pulldowns or rely on internal ones.

4. USB interface (J1, C1–C2, R1–R2)

• VBUS usage: Currently 5 V is only on J1; extend it to the charger and any other circuits needing it. Consider adding input reverse-polarity or surge protection if needed.
• Signal conditioning: 47 pF shunts on D± may be aggressive; ensure they meet USB FS requirements once you know trace lengths. No glaring mistakes otherwise; CC resistors (R1/R2) look correct.

5. Battery monitor / I²C bus (J5 “MCP3021”, R7, J6 connector)

• Analog input over-voltage: MCP3021’s AIN must sit between VSS and VDD.[^mcp-ain] Here AIN (pin 2) is tied to batt while VDD (pin 4) is 3.3 V, so AIN can reach 4.2 V and violate the spec. Fix by adding a resistor divider (and optional RC) or powering the ADC from the battery and level-shifting I²C.
• Add I²C pull-ups: Nets N$13/N$14 lack pull-up resistors. Provide ~4.7 kΩ to 3.3 V so both the MCP3021 and the OLED connector (J6) have a defined bus idle level.
• Bus voltage domains: If you decide to run MCP3021 at battery voltage, include appropriate level shifting to protect the ESP32 pins (absolute max VIO ≈ VDD + 0.3 V).[^esp-io]

6. User inputs & connectors (SW3/SW4, J2, J6)

• Buttons simply short pins to 3.3 V; ensure firmware enables the needed pulldowns (or add resistors) to avoid floating inputs. Otherwise no electrical issues spotted.
• Programming header (J2) and OLED connector (J6) look consistent with shared I²C/SW nets.

Next steps

• Rewire U2 VIN/CE to USB 5 V, add the required 1 µF caps on VIN and VBAT, and confirm your intended charge current.
• Substitute the correct 3.3 V LDO variant and place the mandatory input/output capacitors; add the ESP32 bulk/decoupling caps.
• Protect the MCP3021 input (divider) and add I²C pull-ups; reassess level compatibility if the ADC runs from the battery.
• After these fixes, rerun ERC/DRC before moving to layout.

Addressing the points made:

1. I am purposefully under-charging the battery. It is 30mAh and a 500mAh charge would be charging it at 17C - for an unknown Chinese battery normally rated at 1C. The watch charging in 30 minutes (2C) is perfectly fine.
2. I used the 1.2V footprint because it is the same footprint and pinout as the 3.3V part, but KiCAD did not have it by default. The input/output capacitor pins was a calculated risk, but I think we can bring those back in the next revision.
3. I’m not concerned about the ESP32 setup as I have tested this within an inch of it’s life.
4. The signal conditioning I know to be fine as I have used this with other circuits previously. What is a concern is that the USB 5V is not in use, which I checked, and it’s true!

And what’s worse, is that it should be!

# monitor.py
power = {
  "batt": v_batt,
  "5v": v_5v,
  "3v3": v_3v3,
  "0v": v_gnd,
}
# ...
man = MCP73812T_420I_OT(power)

# mcp73812t_420i_ot.py
if "5V" in power :
  self.man[4] += power["5V"]

Can you spot it? It’s defined as 5v in the power net and 5V in the check! It fails silently! Making it raise an exception if the label is not found immediately shows the issue:

# mcp73812t_420i_ot.py (revised)
if not "5V" in power :
  raise Exception("No 5v found!")
self.man[4] += power["5V"]

  File "/env/src/shared/mcp73812t_420i_ot.py", line 33, in __init__
    raise Exception("No 5v found!")
Exception: No 5v found!

Note: The first version will need to somehow get 5V from the USB bus, otherwise it will not be able to charge the LiPo. There is no way to easily achieve this, so I will need to have a think about it.

5. I believe I should be able to use the ESP32’s pull-ups, and that the OLED display itself has weak pull-ups. That said, the MCP3021 poses a massive issue. I saw the values I wanted to see in the datasheet, and completely missed the values I did not want to see.

The absolute limit is VDD + 0.6V, so 3.3V from the regulator, bringing us up to 3.9V. 3.9V to 4.2V is not too crazy, maybe we could get away with it?

This all said, I wanted to check if there are any known side effects of missing the input/output capacitors:

“What can be the consequence of missing the 3.3V input/output capacitors? It is believed that the ESP32-S3-WROOM-1 module has it’s own internal capacitors, and that the capacitance of the board is sufficient?”

Leaving out the regulator’s 3.3 V input/output capacitors is risky:

• Regulator stability: The TLV740 family explicitly requires at least 1 µF of ceramic capacitance on both VIN and VOUT for loop stability and transient response (§7.1.2, TI TLV740 datasheet SBVS401B). Without them the LDO can oscillate, overshoot, or momentarily drop out, which will show up as random brown-outs or crashes of the ESP32, OLED, and ADC.
• Transient/PSRR degradation: Even if the circuit seems to work on the bench, load steps from the ESP32’s radios or display can inject noise back into the rail. Missing capacitors greatly reduce PSRR and worsen line/load transients, so RF performance, ADC readings, and I²C integrity suffer.
• Module expectations: The ESP32-S3-WROOM-1 reference design still places bulk (≈22 µF) and high-frequency (0.1 µF) capacitors right at the 3V3 pin (Fig. 7, esp32-s3-wroom-1 datasheet v1.4). The module’s internal bypassing only covers on-module circuitry; it assumes the host board supplies its own local decoupling.

In short, rely on the datasheet values—place the required ≥1 µF ceramics at the regulator pins (plus the usual 10 µF/0.1 µF pair near the module). Skipping them invites instability and erratic behavior.

Okay, okay, I will put some capacitors in the new design.

And lastly, what the consequences are for running the ADC out of spec:

“MCP3021 analogue input can be VDD + 0.6V, where VDD should be 3.3V and the maximum will be 3.3V + 0.6V = 3.9V. The battery fully charged is 4.2V. What will happen to the IC?”

According to the MCP3021 datasheet, the analog input pin (AIN) must always stay within VSS –0.3 V to VDD + 0.3 V (see “DC Electrical Specifications,” DS20001805C, p. 3). With the device powered at 3.3 V, any voltage above about 3.6 V violates that limit. Driving AIN with the full 4.2 V from a freshly charged Li‑ion cell forward-biases the input’s internal ESD/clamp structures. In practice that:

• Forces current through the protection diodes into the 3.3 V rail, which can pull VDD above its nominal value, upsetting the ESP32 and other loads.
• Produces gross conversion errors because the sampling capacitor is no longer referenced correctly.
• Risks long-term damage: repeated or sustained over-voltage can overheat the clamp diodes or core circuitry, eventually destroying the ADC.

You should scale or buffer the battery voltage so AIN always remains within the allowed range—e.g., a resistor divider sized to keep the maximum input ≤3.3 V (or add a buffer/level shifter if you need very high input impedance), and optionally include a small series resistor to limit any fault current.

Well, the usable range is actually limited to within VDD + 0.3V… Maybe it could still be used to detect low battery voltage?

Ah, and herein is the problem - those damn diodes. If we drive the analogue input too high, then it will try to drive VDD. Damn.

Note: The ADC cannot be populated in the first design.

Hmm, I think I will be looking for a new ADC in the next design, or I need to be really clever about reliably measuring the 4.2V. To measure 4.2V with this current IC I would need to drive VDD with at least a steady 3.9V, damn.

6. The feedback is fine.

Conclusion

Is using an LLM for netlist inspection worth it? It seems so! I have no idea how many hours this may have saved me in debugging time!

Hopefully all of the parts arrive soon enough and the first revision can be made.

Future

It looks as if micropython make using the OLED display quite easy with the ssd1306 library, offering a framebuffer and some handy drawing functions.

I may consider bringing back the pt-draw library to draw lightweight vector graphics. Maybe I could try parsing some sub-set of the SVG library via the micropython XML library xmltok (or perhaps uxml2dict to get a dictionary) and just support line segments (makes designing in external editors easier).

I still need to figure out how to communicate with the ADC to read the battery level, but this should be quite simple via I2C. It’s important that it can be read, though, as it will be a large part of the power saving strategy.

The next step will be to build out the firmware development and deployment system to make my life easier.

Fixing The ADC

At the time of writing this part, everything is already sent off!

In a future revision, there is a viable path to bring the cost down. After a chat with the people at the local hackspace, the best approach is likely a resistor ladder, but via a MOSFET. That circuit looks like so:

This is with the DMN62D0U-7, an 8p IC. I calculated the MOSFET Beta to be at 0.8 (R_DS = 2.5, V_GS = 1, Threshold  = 0.5). In the simulation (with resistors all set to 10k) the off-current is ~200nA and the on-current is ~700uA, which seems quite acceptable.

Compiling TCC

For the first test, we compile TCC and compare the final binary size and how long it took to compile. The TCC binary was chosen because it’s a relatively large project and it is written in compatible C.

clang, gcc and tcc produce similar binary sizes, but fil-c produces are greatly larger binary.

Similarly, fil-c takes significantly longer to compile, whilst tcc enjoys a reduction in compilation time.

fil-c is likely to be bloated due to the memory safety features, so it really depends on how much you value those. For roaring performance, it does seem to not do as well.

Compiling Links

For the next test we compile Links, as it was one of the tests that the TCC author used to benchmark the performance of TCC. In Fabrice Bellard’s test, he got the following performance figures:

For binary size, each is relatively similar, but it is strange how compiling options appear to make little or no difference to tcc or fil-c.

For compilation time, tcc appears to lose the edge that it had, and fil-c appears to be somewhat consistent.

Compiling u-database

For the next test we compile and execute the performance test for u-database, a flat-binary database key-value store written in C. It will happily read and write hundreds of thousands of records a second (sub microsecond per transaction). In this high-performance application, small changes in compiler and flags can be felt.

When compiling my u-database library, there appears to be no real binary size change for each compiler.

The optimisation flags do appear to hit gcc and fil-c strangely enough, otherwise there is an unexpected uniformity across the software.

With the execution time, it for some reason appears that the longer it took to compile, the longer it also takes to run.

Compiling smollibs

For the next test we compile a series of single-file headers from a project called “smollibs”. Each of them is a tiny C99 library with a basic example test program.

For the next graphs, we will evaluate them together…

There appears to be a “striping” affect on the graphs, particularly around the -Ofast flag. Looking at the GNU explanation for the flags:

-Ofast

Disregard strict standards compliance. -Ofast enables all -O3 optimizations. It also enables optimizations that are not valid for all standard-compliant programs. It turns on -ffast-math, -fallow-store-data-races and the Fortran-specific -fstack-arrays, unless -fmax-stack-var-size is specified, and -fno-protect-parens. It turns off -fsemantic-interposition.

It doesn’t appear to specifically be the -O3 flag as we also test this, so it’s safe to assume that one of the other flags causes the binary size to increase.

Rather than making a case for a specific compiler, these tests seem to be making a case for testing which compiler specifically works for your use-case!

Compiling rclc

For the last comparison, I wanted to see how well each compiler performed when compiling rclc, the C-based wrapper for ROS2. The first issue is that there is not an easy-to-point-to file that represents the output of the compilation process, colcon hides files away and I kind of ran out of time.

clang and gcc gave relatively predictable compile times over all of the test flags, but tcc failed to compile the project unfortunately. I believe it is because it is a C99 based compiler, whereas rclc uses more modern C features. fil-c also fails to compile, but I didn’t have time to try and set it up correctly.

Conclusion

Should you switch to tcc as your daily driver C compiler? Probably not, but it’s not as terrible an option as you may have original considered! But it will produce valid C code for many projects, it is mostly faster to compile, and the execution time is comparable. For speed or memory restricted applications, you may very well consider tcc as a valid option!

In the future it would be good to get rclc working correctly, and to compare libc implementations. Additionally it is important to consider shared vs static library building - I suspect that we see a move towards static libraries as disk size is cheap, RAM is plentiful, and the cost of sharing memory across increasing numbers of cores is more heavily felt.

Background

Both neural networks and GP take some input x, and produce some output y, using some learned model M. We can consider this to be M(x) = y. There is then some fitness/error function that is used to adjust the model M to better output the result.

The likes of LLMs are essentially just predictive models, where given some current state s_t, predict a future state s_t + 1, i.e M(s_t) = s_t + 1. An agent can explore a predictive model by proposing an action a_t, i.e. M(s_t|a_t) = s_t + 1. Some more advanced agents may consider the result of following several future actions A_t, i.e. M(s_t|A_t) = s_t + 1 - the computation explodes for large action sequences.

This compute required can be extremely large for training and recall, so before this really became feasible on better computers, researchers developed reinforcement learning. The major idea was not to predict a future state s_t + 1, but instead the best future action a_t + 1, i.e. M(s_t) = a_t + 1.

Structure

A large breakthrough in deep learning was figuring out how to build these deep structures. AlexNet arguably opened the door to these deeper structures, but it took a while to get there.

Given how important work on structures has been, it seems crazy that we do not lean into GP for designing and building these structures. Given how large these models are, what we need is either a representative simplified/reduced model, or a reduced computation training/evaluation.

I believe it is worth noting that whilst GP can be used for the structure, that learning for weights, activation energies, etc, are still far more efficiently addressed by the current learning techniques.

This could be an interesting place for future work…

Intrinsically Motivated GP Agents

What I want to revisit is a GP-based agent where a fitness/error function is not strongly defined. This would be a model in the form M(s_t|A_t) = s_t + 1 where an algorithm searches over the predictive model and produces a_t + 1.

What, if, we setup our search that we look for the intrinsic motivation algorithm itself? Possibly a bigger question yet, how would we know we found an interesting algorithm? Do we rate survival, exploration behaviour, collaboration with other agents, the ability to terraform the environment, the ability to approximate these measures cheaply? By what measure do we consider the agent successful?

The answer is: I’m not sure. But I think we’re asking the right questions.

Inspiration

Whilst searching on the wild world web I found this article by Andrei Ciobanu on Implementing Hash Tables in C. He kindly offers his source code which was an inspiration for my testing.

Unfortunately I was unable to use his code or testing, as he was interested in handling hash functions that return uint32_t and I am interested in char*, as well as wanting to visualize the quality of the hashes and benchmark the performance. I therefore took the time to write my own tests.

Testing Previous Functions

I created a small piece of checking code here. It outputs results, as well as various visualisation in PGM format. There is then a script to convert the PGM files to scaled JPG files (which are displayed here). The idea is to be able to visualise the collisions of the hash function, and therefore the quality. Black areas mean large numbers of collisions, whereas white means lower numbers of collisions.

In these tests, I test the collisions for uint8_t, uint16_t and “uint24_t” (which doesn’t exist). This in turn gives us single-byte and two-byte collisions, whereas the three-byte collisions indicate the hashing quality after the initial concentration area.

Why not test more bytes for collisions? The problem is storing the collisions in RAM. These tables are already quite large! The program already consumes in excess of 500MB of RAM during testing and each additional bit to be tested doubles that!

Function 1

This was my initial “make something work” function. It did an ‘okay job’ during initial testing.

char* hash_cust1(char* s, char* d, int n){
  int x = -1;
  while(s[++x]) d[x % n] = s[x] ^ (x < n ? '\0' : d[x % n] << 2);
  while(x < n) d[x++] = '\0';
  return d;
}

Performance: In terms of performance, this was my second best forming algorithm. We pay heavily for the inline ternary operator and this for sure affects branch prediction.

Collisions: This was awful.

As can be seen in the one-byte output, the function was completely unable to hash char characters with large overlap.

The two-byte output shows how insanely bad this problem is, with collisions typically happening in exactly the same locations.

It obviously doesn’t improve as we look towards three-byte output. I late realised it is even worse than I thought. If |s| ≥ n, it overrides any previous good effort attempt to hash the first two bytes. Ouch!

Function 2

At this time I has realised that my hashing function was significantly holding back my program, so I looked to make some small changes to improve test results.

char* hash_cust2(char* s, char* d, int n){
  int x = -1;
  while(s[++x]) d[x % n] = s[x] ^ (s[x + 1] << 3);
  while(x < n) d[x++] = '\0';
  return d;
}

Performance: We managed to lose the ternary operator, so things were looking good! But - for some reason, this actually decreased performance. I imagine there was some cost to looking one byte into the future and performing some operation on it with the current byte. Perhaps there is some cost associated with accessing memory more than once. I would imagine that s[x + 1] would be kept in a register for the next loop, but clearly not?

Collisions: It’s still not great, but clearly a massive improvement!

As we can see, the one-byte output is now more spread out! It’s still a very repeatable, not very well spread pattern - but this is a remarkably better distribution.

Unfortunately this didn’t hold up as we look at the two-byte output. It’s still a much better distribution, but not great.

One thing this function does well is to scatter the distribution across the entire range. This is where we see that s[x + 1] come into its own - but again it is too costly.

Function 3

It’s now that I decided to focus more heavily on the entropy of those first two bytes, where here is achieved by having a dedicated line just for hashing those first two bytes repeatedly.

char* hash_cust3(char* s, char* d, int n){
  int x = -1;
  while(s[++x]){
    d[x % n] = s[x];
    d[x % sizeof(uint16_t)] ^= s[x] << 3;
  }
  while(x < n) d[x++] = '\0';
  return d;
}

Performance: As a result of this extra dedicated operation, we lose performance. For the purpose of the application though, this was worth it, as it reduced collisions and therefore increased overall performance.

Collisions: Collisions are tonnes better in this version! It’s night and day!

As you can see, for one-byte we have a much better spread, but still not perfect.

This is also the best two-byte output (so far) as we now see

Function 4

This is the ‘best’ function I could derive based on function 3 (cust3). It now uses memset to zero out the memory, which means a bunch of optimizations methods can be used to accelerate the zeroing of this space. Additionally everything is in-lined to increase simplicity.

char* hash_cust4(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s; ++x) *(int16_t*)d = (*(int16_t*)d << 5) ^ *(int16_t*)d + (d[x % n] ^= *s++);
  return d;
}

Performance: The performance is better than function 3 (cust3), but not as as good as the functions before.

Collisions: The number of collisions are reduced, but not significantly. You can see there is a very visible pattern in the visualisation and this this will ultimately cause larger numbers of hash collisions.

The one-byte spread clearly has a pattern, but this is significantly better than previous attempts.

The two-byte pattern is clearly much better than all previous attempts - we can now be certain that hashes are distributed over the entire space.

The thee-byte visualisation shows that we still suck at distributing the data over multiple bytes. The entropy is entirely concentrated on the first two bytes.

Testing New Functions

Ideally, we want a faster function that reduces collisions with the exact same data. Is this possible? Yes.

Armed with this new knowledge, I now test some new functions (and display the results for previous algorithms during the same run window) for 2²⁴ keys:

As you can see, cust12 and cust13 have the least hash collisions whilst maintaining respectable timing (compared to the cust4 benchmark).

And for the purpose of really exaggerating the timing, I run them also at 2²⁸ keys (268435456):

As you can see, cust5, cust7, cust8, cust12 and cust13 all outperform cust14. We’ll now go into a little more detail about the algorithms and how they were designed…

Function 5

This is similar the function 4, but operates on the theory that we are shifting our entropy out the front of the uint16_t. Instead of shifting by 5, we instead shift by 3:

char* hash_cust5(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s; ++x) *(int16_t*)d = (*(int16_t*)d << 3) ^ *(int16_t*)d + (d[x % n] ^= *s++);
  return d;
}

Shifting by lower values seems to reduce the number of clock-cycles required to do the shift. Ideally shift values should be as low as possible.

Performance: As can bee seen, we reduce the running time when compared to function 4 (cust4).

Collisions: We perform slightly worse on collisions, but it is worth the speed-up.

The one-byte pattern looks very good. No place in particular looks overly concentrated.

This scales well to the two-byte pattern where we see a relatively nice distribution. At this point I realised that the blocky-ness acutally comes from that XOR operation which can be addressed in the future!

Three-byte pattern still have really poor hash distribution and is causing tonnes of collisions.

Function 6

With function 6 (cust6) I experimented with using rotl16 instead of shift, my theory being that we were shifting entropy straight off the bit.

char* hash_cust6(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s; ++x) *(int16_t*)d = rotl16(*(int16_t*)d, 5) ^ *(int16_t*)d + (d[x % n] ^= *s++);
  return d;
}

Performance: Using the in-line function took a massive performance hit!

Collisions: It made only a little difference to the number of collisions ultimately.

In the one-byte pattern you can clearly see some blocking - very interesting! There is clearly something going on here!

This is not a bad distribution at all for the two-byte distribution.

The three-byte pattern still sucks, argh!

Function 7

At this point, I realised that we could overcome the issue of losing bits out the front of the bit shift simply by using larger integers instead of rotation:

char* hash_cust7(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s; ++x) *(int32_t*)d = (*(int32_t*)d << 3) ^ *(int32_t*)d + (d[x % n] ^= *s++);
  return d;
}

Performance: It’s obviously much faster to use larger integers rather than the dedicated rotation function.

Collisions: We saw a nice little reduction in the number of collisions, awesome!

The one-byte pattern looks quite nice again.

The two-byte pattern is still a little blocky.

The three-byte pattern still sucks…

Function 8

This was just an experiment in performing the ++x elsewhere.

char* hash_cust8(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s;) *(int16_t*)d = (*(int16_t*)d << 5) ^ *(int16_t*)d + (d[++x % n] ^= *s++);
  return d;
}

Performance: Runs a little slower.

Collisions: A regression in reducing collisions.

Nice one-byte pattern.

A little blocky two-byte pattern, but okay-ish.

Again an awful three-byte pattern.

Function 9

At this point I was getting a little desperate, the three-byte distributions have so-far sucked! I started experimenting with purposely ensuring that the upper hash was definitely touched by our algorithm:

char* hash_cust9(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s;){
    *(int16_t*)d = (*(int16_t*)d << 5) ^ *(int16_t*)d + (d[++x % n] ^= *s++);
    *(int16_t*)(d + (x % n - 1)) ^= *(int16_t*)d << 3;
  }
  return d;
}

Performance: Doing too major calculations rather than one of course had a massive impact on performance.

Collisions: This is the first time we have seen such a massive reduction in collisions!

One-byte distribution doesn’t look all so bad.

Two-byte distribution is a little blocky, but not all so bad.

The three-byte distribution is… Wait a second! That’s awesome! We have a nice spread over the entire 3-byte distribution space!

Function 10

So at this point, I needed to think of a completely different approach. I turned back to my original resource for simple hash functions and thought it would be worth also checking out the sdmb implementation, specifically the gawk implementation:

char* hash_cust10(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s; ++x)
    *(int32_t*)d = (d[x % n] ^= *s++) + (*(int32_t*)d << 6) + (*(int32_t*)d << 16) - *(int32_t*)d;
  return d;
}

Performance: A reduction in performance when compared to function 9 (cust9). Still not as good as function 4 (cust4), but I’ll take it!

Collisions: Another reduction in collisions! I believe the merit here is we see multiple ‘smears’/‘spreads’ of data across the range.

The one-byte pattern looks super cool!

That two-byte pattern is quite evenly distributed too, not bad!

And look at that three-byte pattern, that’s very nice!

Function 11

We speed-up function 11 (cust11) by changing the order of the operations (long story):

char* hash_cust11(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s; ++x)
    *(uint32_t*)d = (*(int32_t*)d << 16) + (*(int32_t*)d << 6) + *(int32_t*)d + (d[x % n] ^= *s++);
  return d;
}

Performance: This gives us a little speed-up.

Collisions: Collisions are for some reason slightly higher, although the algorithm should be essentially the same for the most part.

Weird one-byte spread.

Weirder two-byte spread.

Nice three-byte spread.

Function 12

Instead of bit-shifting with (*(int32_t*)d << 16), we now just cast instead with *(int16_t*)(d + 2). We also use 16 + 5 bit shifting rather than 16 + 6 for a slightly nicer distribution:

char* hash_cust12(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s; ++x)
    *(uint32_t*)d = *(int16_t*)(d + 2) + (*(int32_t*)d << 5) + *(int32_t*)d + (d[x % n] ^= *s++);
  return d;
}

Performance: For our effort we get significant speed-up, out-competing funct4 once again!

Collisions: Collisions are also reduced now owing to the shifting of 16 + 5.

A very nice one-byte distribution.

And a very nice two-byte distribution to match.

The three-byte distribution, whilst not the prettiest to look at, is much better than previous attempts.

Function 13

This function is a very small iteration of the previous function.

char* hash_cust13(char* s, char* d, int n){
  memset(d, 0, n);
  for(int x = 0; *s;)
    *(uint32_t*)d = *(int16_t*)(d + 2) + (*(int32_t*)d << 5) + *(int32_t*)d + (d[x++ % n] ^= *s++);
  return d;
}

Performance: Slightly better performance.

Collisions: The same number of collisions. The below exist just for completeness.

Final Thoughts

I’m pretty happy with the final implementation and believe is is a significant improvement over previous attempts. That said, I still believe there are improvements that can be made in this space:

1. Single byte (uint8_t) addressing - Removing the dependence on large integers is ideal. Having a hash function that can be used on any architecture (from an 8 bit microcontroller to a 64 bit server) and produce the same result is clearly beneficial.
2. Remove the memset - Ideally we do the wiping as we setup the hash.
3. Spread bytes over the entire hash - Even for short keys, we should be able to sample any part of the hash and get a pretty decent distribution.

In future experiments I will look to sample the end of the hash too when looking at the distribution. I suspect they all currently absolutely suck at filling this space in which anything and we get tonnes of collisions out there.

If you make improvements based on these tests, please let me know! Better hashes are better for everybody!

Requests

So far I have tracked 121,418 unique requests since the 18th July 2020 till today, 23rd October 2020. I haven’t tracked the error log as this was a trial.

As you can see, I have slowly crept from about 500 requests per day to about 1,000 requests per day (double) in about 3 months. This is zero search engine optimization (SEO), zero advertising and little up-keep. Hell, it only supports HTTPS because it has to.

Locations

Where are the requests coming from?

It seems they are mostly coming from the United States, followed next by Singapore, China, Netherlands, France, Germany and New Zealand. I have no idea why these particular Countries, but it’s an interesting breakdown given I don’t have many friends in the US I’ve given my site details to! Organic growth perhaps?

In 2017 I did a little experiment to figure out who was attacking a server I was running based in Russia - which looks greatly different from these logs. China featured massively in those.

Future

So going forwards I want to start tracking a few more things:

• Errors (i.e. invalid requests) - this should indicate people trying to attack the server.
• SSH port - who is trying to log into my server?
• Other ports - What else is being attacked?

Background

I use a custom script to pull my nginx logs, manipulate them and then graph the results for the website. It is responsible for a few tasks:

1. Backup Dead Social - Make a clone of Dead Social files ¹.
2. Get Logs - Copy the logs from the server via scp.
3. Merge Logs - Merge the logs into a single file to be processed.
4. Make Logs Unique - Remove non-unique log entries.
5. Abstract Log Data - Abstract data of interest, ready for graphing.
6. Create Access Graph - Generate the first graph.
7. Create Location Graph - Generate the second graph.
8. Clean-up Temporary Files - Remove the temporary files we made along the way.
9. Display Nginx Data - Use an old nginx-parser program to process and visualise the logs.

This system has been getting slower over the years, to the point where it now takes the best part of a day to process the logs. This both uses significant system resources and means that I cannot access important data for quite some time.

How large is the data?

$ ls -lh storage.txt
1.5G storage.txt
$ wc -l storage.txt
7100816 storage.txt

1.5 GB and 7 million entries being processed by an old bash script. Yikes. Time to do some debugging.

Old Processing

The first task was to add some timing code:

# Time variables
time_beg=0
time_end=0
time_run=0

# time_start()
#
# Begin timing.
function time_start {
  time_beg=$(date +%s.%N)
  time_end=$time_beg
  time_run=0
}

# time_start()
#
# Begin timing.
function time_stop {
  time_end=$(date +%s.%N)
  time_run=$(echo "$time_end - $time_beg" | bc -l)
  echo "$time_run"
}

Now knowing how long one or more items takes to process is as simple as:

time_start
task_to_run
time_diff="$(time_stop)"

I wrapped the main functions and timed how long they take:

“nginx_parser_rt” is interactive and just represents how long I left the computer to sit there in the background. “logs_abstract_rt” represents 4.7 hours of straight processing. Oufe.

Re-Write Processing

The old processing code was as follows:

while read p; do
  IFS=' ' read -r -a cols <<< "$p"
  ip="${cols[0]}"
  country="$(geoiplookup $ip | cut -d " " -f 4)"
  country=${country::-1}
  timestamp="${cols[3]:1}"
  date="${cols[3]:1:-9}"
  time="${cols[3]:13}"
  method="${cols[5]:1}"
  path="${cols[6]}"
  response="${cols[8]}"
  echo "$date" >> out-date.txt
  echo "$country" >> out-ip.txt
done <storage.txt

Line by line, we process each of the columns and pull out the important data. My thinking was that we may be paying for double-handling data, so I re-write this to the follows:

# Clean data
while IFS=' ', read ip z0 z1 timestamp zone method path type response size; do
  country="$(geoiplookup $ip | cut -d " " -f 4)"
  country=${country::-1}
  timestamp="${timestamp:1}"
  date="${timestamp::-9}"
  time="${timestamp:13}"
   zone="${zone::-1}"
   method="${method:1}"
   type="${type::-1}"
  echo "$date" >> out-date.txt
  echo "$country" >> out-ip.txt
done <storage.txt

And the results of that is as follows:

It helped a little, but clearly we have to get a little more creative here…

C Implementation

I suspected I’m mostly getting screwed by the geoiplookup command, that loads a database of a few MB and then looks up the full locations details, which I then have to filter out.

A better system is to load the database (of just a few MB) into RAM and then keep performing the searches against this for only the Country code. Additionally if we process the all of the logs in C++ we further save time.

I decided to use the maxmind’s geoip-api-c, which up until now, has worked great.

//#define GEOIPDATADIR "geoip-api-c/data/"
#define GEOIPDATADIR "/usr/share/GeoIP/"
#define PACKAGE_VERSION "1.6.12"

If you try to use the database yourself, you will likely need to declare these variables. I had no end of trouble trying to link to the development library and I suspect it is broken since quite a while. It was much easier to directly include the files.

Note that the value for GEOIPDATADIR if for my system, which I found through man geoiplookup. On Linux the location appears to be: /usr/share/GeoIP/. Technically there is a file in the repository at geoip-api-c/data/ - but this did not work for me and was far too small. There is an update tool, but this requires you to register ².

#include "geoip-api-c/libGeoIP/GeoIP_deprecated.c"
#include "geoip-api-c/libGeoIP/GeoIP.c"
#include "geoip-api-c/libGeoIP/GeoIPCity.c"

One little trap here is not including GeoIP_deprecated.c.

#include <stdarg.h>
#include <stdbool.h>
#include <stdio.h>
#include <sys/time.h>

#define PARSER_VERSION "0.0.1"
#define PARSER_LINE_LEN (256 * 256)

We might need a larger line length if we expect to have someone significantly testing our server. Most servers and browsers will allow URLs of 65k. In theory, each of those fields could be significantly large and this would cause issues with our parser.

/* Source: https://stackoverflow.com/a/51336144/2847743 */
int64_t currentTimeMillis(){
  struct timeval time;
  gettimeofday(&time, NULL);
  int64_t s1 = (int64_t)(time.tv_sec) * 1000;
  int64_t s2 = time.tv_usec / 1000;
  return s1 + s2;
}

Some timing code that repeatedly proves to be useful.

int flog(const char* format, ...){
  va_list args;
  va_start(args, format);
  fprintf(stderr, "[%li] ", currentTimeMillis());
  vfprintf(stderr, format, args);
  fprintf(stderr, "\n");
  va_end(args);
}

Nice little wrapper around printf() to allow me to automatically add a timestamp to the front of logs.

/**
 * process()
 *
 * Process the logs.
 *
 * @param raw The raw nginx logs.
 * @param date The location of the date file.
 * @param date The location of the location file.
 **/
void process(char* raw, char* date, char* loc){
  /* Setup buffers */
  char buff[PARSER_LINE_LEN + 1];
  /* Setup files */
  FILE* fraw = raw != NULL ? fopen(raw, "r") : NULL;
  FILE* fdate = date != NULL ? fopen(date, "a") : NULL;
  FILE* floc = loc != NULL ? fopen(loc, "a") : NULL;
  if(fraw == NULL){
    flog("Could not open '%s' for parsing", raw);
    return;
  }
  /* Setup GeoIp */
  GeoIP* gi = GeoIP_open(GEOIPDATADIR "GeoIP.dat", GEOIP_MEMORY_CACHE | GEOIP_CHECK_CACHE);
  /* Loop over the file */
  while(fgets(buff, PARSER_LINE_LEN, fraw)){
    int x = 0;
    char* ip = NULL;
    char* date = NULL;
    char* time = NULL;
    char* path = NULL;
    char* t = strtok(buff, " ");
    while(t != NULL){
      switch(x++){
        case 0 :
          ip = t;
          break;
        case 1 :
        case 2 :
          /* Do nothing */
          break;
        case 3 :
          date = t + 1;
          t[12] = '\0';
          time = t + 13;
          break;
        case 4 :
        case 5 :
          /* Do nothing */
          break;
        case 6 :
          path = t;
          break;
        default :
          /* No more of interest, quit early */
          t = NULL;
          break;
      }
      /* Check if we can stop tokenizing early */
      if(t == NULL){
        break;
      }else{
        t = strtok(NULL, " ");
      }
    }
    if(ip != NULL && floc != NULL){
      char* code = GeoIP_country_code_by_addr(gi, ip);
      fprintf(floc, "%s\n", code);
    }
    if(date != NULL && fdate != NULL){
      fprintf(fdate, "%s\n", date);
    }
  }
}

A few tricks going on here. Firstly note the use of the strtok() function that creates tokenized strings in-place. The second trick is that we only grab the interesting data and we drop everything else as soon as possible. We try not to waste cycles processing data we have no plan to use.

/**
 * help()
 *
 * Display program help.
 *
 * @param argc Number of arguments.
 * @param argv Pointer to the arguments.
 * @param x The current offset into the arguments.
 * @return The new offset into the arguments.
 **/
int help(int argc, char** argv, int x){
  flog("./parser [OPT]");
  flog("");
  flog("  OPTions");
  flog("");
  flog("    -d  Append data to a date file");
  flog("          <STR> Location of date file");
  flog("    -h  Display this help and quit");
  flog("    -l  Append data to a location file");
  flog("          <STR> Location of location file");
  flog("    -r  The raw nginx logs to be read");
  flog("          <STR> Location of nginx logs");
  flog("    -v  Display version information");
  return x;
}

/**
 * main()
 *
 * Process the command line arguments.
 *
 * @param argc The number of arguments provided.
 * @param argv The arguments provided.
 * @return The exit status of the program.
 **/
int main(int argc, char** argv){
  static char* raw = NULL;
  static char* date = NULL;
  static char* loc = NULL;
  for(int x = 0; x < argc; x++){
    if(argv[x][0] == '-'){
      switch(argv[x][1]){
        case 'd' :
          date = argv[++x];
          break;
        case 'h' :
          x = help(argc, argv, x);
          return EXIT_SUCCESS;
        case 'l' :
          loc = argv[++x];
          break;
        case 'r' :
          raw = argv[++x];
          break;
        case 'v' :
          flog("Version: %s", PARSER_VERSION);
          break;
        default :
          flog("Unknown option '%s'", argv[x]);
          break;
      }
    }else{
      flog("Unknown parameter '%s'", argv[x]);
    }
  }
  if(raw != NULL){
    process(raw, date, loc);
  }else{
    flog("Require raw logs to be processed");
    return EXIT_FAILURE;
  }
  return EXIT_SUCCESS;
}

The result?

A 383x speed-up (or 38,268% speed-up) - not bad at all!

Server

Encase you are wondering, this is the current server statistics:

The reason I needed to have this better processed is that my server now hosts static files for another business and I need to ensure the server resources stand up well. If I suddenly start seeing significant increases in traffic I may consider upgrading the server to accommodate.

Idea

I was chatting with a friend about completely removing JS from the web entirely and coming up with the various conditions that would or wouldn’t work. One of the ideas was to build a lightweight browser that supported HTML and CSS features, but did not allow even a single line of JS to run. This is what we come up with:

• Social media would survive. You would have to say goodbye to infinite scrolling, but for the most part the experience would be recognizable. As long as you’re doing authentication via cookies over TLS, it should be a smooth experience. There is for example mbasic that I personally use to access Facebook - it breaks all the time but actually much less than letting their crappy scripts run in your browser 24/7.
• Banking websites should be okay for the same reason. I suspect the use of JS in banking software has actually caused more security issues than it has ever resolved. For example, there have been multiple times a banking site have been compromised by having a CDN changed under them.
• Chat applications can work, but of course it currently requires manual reloading. That automatic loading of new content simply won’t work currently.
• Many niche cases, like running web torrents, etc. Given how many niche cases there are, this would be a small minority of the web. Generally, most of these could probably be re-worked into a desktop application or run server side.

Feature

So the feature I am thinking of is a “automatic re-loader” capability. It wouldn’t be intelligent or based on some event, but instead time-based with several configuration options. I think the best candidate for this would be an <iframe>, it already has several nice sandboxing aspects and is well supported. In the grand-scheme of things, programs like Dillo or Lynx could be upgraded to support such a feature without too many changes to their codebase.

Essentially I think the HTML would look something like this:

<iframe src="test.html" reload="auto">
</iframe>

So the introduction would be:

This is not without potential security issues:

1. DDoS external resources – Setting the timeout to 1 millisecond and then pointing it to some external resource of a popular server would potentially DDoS the external server. Therefore this should be an opt-in option.
2. DDoS own server – If your server is under a heavy traffic load, you may want to tell the reloader to slow down.
3. Recursive reloading – Potentially you could be asked to load an iframe, that loads an iframe, which… You get the point. If each one of these also has a reload timer, it could really bring your browsing experience down to a crawl.

Considering these possibilities and for the purpose of maintaining backwards compatibility, a web page that offers the ability to reload can throw a comment in the top of the webpage like so:

<!DOCTYPE html>
<!-- @timeout="50" -->
<html>

Note: It would be nicer to have it in the “DOCTYPE” declaration, but I’m unsure what issues this would cause for backwards compatibility.

Only web pages that offer the timeout comment before the <html> tag (or first HTML element) will be allowed to perform the reload. A web server can thereby set this value to anything that is suitable during page generation. The values would be like so:

Conclusion

I think such a web feature would be worth considering. It literally takes basic HTML and makes it interactive via a server, removing the need for quite a bit of JS currently used.

Some potential applications of this feature:

1. IRC in the browser using only HTML (or other chat features)
2. Auto-updating email interface using only HTTP
3. New feeds or status bars for websites
4. Some basic types of games
5. Checking for document changes