> then the definition of the "shell" and "core" becomes problematic in this comparison
I agree -- if you're trying to make the words "shell" and "core" mean the same things between FCIS and GCSS, or identify the same parts of a program, then there will be problems. I think FCIS and GCSS are just two different ways of analyzing a program into pieces. Just as the number 8 can be seen through addition as 3 + 8 and through multiplication as 2 * 4, a program can be analyzed in multiple ways. If you view a program through the lens of FCIS, you expect to see a broad region of the program in which side-effects don't occur and a narrow region in which they do. If you view a program through the lens of GCSS, you expect to see broad parts of the program that solve general problems, and narrower regions in which those problems are instantiated in specific. The narrower regions are all "shell"-shaped, but that doesn't mean they are "the" shell. They have in common simply that they wrap a bulk of functionality to interface it to a larger context.
> At the core, things can be more integrated than at the boundary, and so it can have more business-specific rules.
I tend to disagree. Decomposition is a fundamental part of software engineering: we decompose a large problem into smaller ones, solve those, them compose those solutions into a solution to the large problem (c.f. Parnas' "On the criteria to be used in decomposing systems into modules"). It is often easier to solve a more general problem than the one originally given (Polya's principle). Combining the two yields GCSS.
A solution to each individual small problem can be construed as having its own generic core, and the principles used in composing sibling solutions constitute the specific shells that wrap them, allow them to interface, and together implement a solution to a higher-level problem.
Because there are multiple of these "cores", each solving a decomposed part of the top-level problem, it's hard for me to see how "At the core, things can be more integrated than at the boundary".
> The definition question is, if you take an application, where is the business logic, is it in the "core" or not?
I don't mean to be a sophist, but I think I need a more precise meaning of "business logic" before I can answer this question. In the process of solving successively smaller (and more general) problems, we abstract away from the totality of the business problem being solved, and address smaller and less-specific aspects of that problem. It may be that each subproblem's solution is architected as an individual instance of FCIS, as is often argued for microservice architectures; or that each subproblem is purely functional and only the top-level solution is wrapped in an imperative core; or anywhere in between. Needless to say, I think that choice is orthogonal.
As a result, I would say that the business logic itself has been factorized and distributed across the many subproblems and their solutions, and that indeed the "specific shell"s that are responsible for specializing solutions toward the specific case of the business need may necessarily include business logic. For instance, when automating a business process, one often needs to perform a complex step A before a complex step B. While both A and B might be independently solvable, orchestrating them together is still "business logic", because they need to be performed in order according to business needs.
(In all of this, perhaps you can see why I don't think the "core" and "shell" of FCIS should be identified with the "core" and "shell" of GCSS. Words are allowed to have contextual meanings!)