The "ground" controller manages taxing around the "movement areas" (i.e. taxiways) on the ground. This notably does not include the runways. And it depends how much of the actual ramp and parking area they control (those are sometimes non-movement area and it's the pilots job to not hit anything).
The "tower" controller manages the actual runways, and the airspace within several miles of the airport laterally and a few thousand feet vertically (varying at each airport). This includes sequencing all the planes that want to take off of land there, and everybody maneuvering around that immediate area.
For the large airports that mostly big airliner flights, that sequencing is largely worked out by the approach controllers dozens or hundreds of miles ahead of time. So there's a steady stream of planes following standardized approach procedures at just the right distance apart.
Outside of the ~30 busiest airports in the country though, there is also a lot of general aviation in small planes. They want to transition through that airspace, or do a dozen laps around the "pattern" to practice landings, etc. Even at fairly major airports, there's plenty of GA activity. For example at Burbank, Ontario, John Wayne, Long Beach, San Jose, Oakland, etc in California. It's only really SFO and LAX where that doesn't really happen, because they set fees to shoo the peons away.
SQL is a small but very busy airport that is almost exclusively GA. There are several flight schools there with multiple planes each, and it's sandwiched in complicated airspace between SFO, SJC, OAK, and open bay.
The tower at this kind of airport is doing a delicate dance keeping multiple planes buzzing around in a rectangular pattern all day every day. Some of which are faster than others. Less frequently a larger much faster plane wants to get in or out and they're getting handed off from approach. Helicopters are doing tours and wanting to cross through. And a lot of the people flying are students that are new at this, don't know how to talk or listen on the radio right yet, make mistakes following directions, etc.
With the tower closed, all those people have to coordinate on a party-line radio with each other about where they are, what they're doing, etc to hopefully not hit anyone. So yeah... it's possible, but it's going to be a mess, and that's why tiny airports like this with virtually no commercial passenger service have a tower.
Also if you're leaving the immediate area, someone at the airport (ground, tower, or "clearance delivery", depending) normally will coordinate putting your destination (for visual/VFR flight following) or full route (for IFR/instruemnt) into the ATC systems before you takeoff so that you can talk to the approach controllers once you leave and they can provide you traffic advisories, etc.
With nobody at the tower to do that, you have to "cold call" approach once already airborne. Or if your route allows, just not get flight following at all (and then ATC has no way to reach you). So SQL tower closing will also add to the workload for the SFO/OAK approach area.