I'm processing public budget data with Rakudo v2023.04 (Raku v6.d) using the module Text::CSV 0.012.

The input file has 150000 lines (records) and I try to load just one column (field) with:

my Str $data = slurp $file_name;
my $fh = IO::String.new($data);
my $csv = Text::CSV.new(';', '"');    
my @column = $csv.getline_all($fh).map( *[$target_col] );
[download]

When executing I get:

> raku clean_class.raku PPP_DSP_2002-2021.csv
Flattened array has 150000 elements, but argument lists are limited to
+ 65535
  in method print at C:\workbench\budget\lib\IO\String.pm6 (IO::String
+) line 40
  in method new at C:\workbench\budget\lib\IO\String.pm6 (IO::String) 
+line 26
  in method new at C:\workbench\budget\lib\IO\String.pm6 (IO::String) 
+line 13
  in sub MAIN at clean_class.raku line 77
  in block  at 'SETTING::'src/core.c/Main.pm6 line 421
  in sub RUN-MAIN at 'SETTING::'src/core.c/Main.pm6 line 416
  in block <unit> at clean_class.raku line 15
[download]

As a workaround, I substituted line 40 of IO::String

        @!content.push: |@x;
[download]

with

        @!content.push: $_ for @x;
[download]

and the script runs without complaints.

I'm wondering if some other solution could be feasible?

Thanks and greetings.

In an attempt to update some ancient straight-up DBI methods, e.g., prepare-> and execute->, I'm using a CGI::Application plugin that I've used a hundred times successfully. This time it's returning:

Error executing run mode 'sl': must call dbh_config() before calling dbh(). at [some line].

This is close to the question in this node, but was never answered.

Start by setting up the connection using dbh_config():

use base 'CGI::Application';
use CGI::Application::Plugin::DBH (qw/dbh_config dbh/);

sub cgiapp_init {
    my $self = shift;
    
   # connect to dssubscriber DB
   $self->config_file('/usr/home/[somepath]/conf/subscriber.conf'); # 
+config_file is a C::A::P::C::S method

   my $data_source = sprintf 'DBI:mysql:database=%s;host=%s',
        $self->config_param('mysql.database'),
        $self->config_param('mysql.host');

        $self->dbh_config('subscriber', 
                [$data_source,
                $self->config_param('mysql.username'),
                $self->config_param('mysql.password'),
                { RaiseError => 1 }
                ]);
}
[download]

Here's the sub that uses the connection and throws the error:

sub get_setup {
    my $self = shift;
    my $subscriber_id = shift;    

    my $stmt = 'SELECT * FROM admin_setup WHERE subscriber_id = ?';
    my %setup = % { $self->dbh('subscribers')->selectrow_hashref($stmt
+, undef, $subscriber_id) };
    
    return \%setup;
}
[download]

Data Dump of $self:

'__DBH_CONFIG' => {
                 'subscriber' => [
                                   'DBI:mysql:database=dssubscribers;h
+ost=qs4821.pair.com',
                                   '1045567_6_w',
                                   'jkgda9leUe89',
                                   {
                                     'RaiseError' => 1
                                   }
                                 ]
               },
'__DBH' => {
          'subscriber' => bless( {}, 'DBI::db' )
        }
[download]

I looked at the plugin's code and it looks like this should work:

unless ($self->{__DBH_CONFIG}{$name}){
        __auto_config($self, $name);
        croak "must call dbh_config() before calling dbh()." unless $s
+elf->{__DBH_CONFIG}{$name};
    }
[download]

What am I missing?

As they were created and maintained manually, over the years the descriptions for each code of the economic classification of expenditures and revenues were becoming polluted. For example, one year the description for code 20 is "Automotive repairments", for other year the same code had "Auto repairs", for other year it is "Vehicle maintenance", and so on. Although most of the times the descriptions match, there are differences between years. Not just word differences, also abbreviations, accents, lower- uppercase, etc...

Unfortunately all the values for one field (column) are composed of the concatenation of the code and the description, e.g. "20.- Vehicle maintenance". There aren't two separate fields for code and description. This way it is hard to create pivot tables and such things by people who don't know how to program.

Task

Normalize the values (strings composed of code and description) of certain (not all) fields. Write a program showing to the user by year all the strings for which the associated code repeats at least once. Also, as a suggestion present the most recent (by year) code+description string (assuming it is the "best", more accurate, more descriptive,...). Let the user interactively choose from all the options shown. Once finished, write out a CSV file with just one column containing the normalized values. This file can then be used to easily replace the whole column in the original input CSV file by using a spreadsheet app, like LibreOffice Calc or MS Excel.

Example session (target column 12):

$ raku clean_class.raku -t=12 PPP_INC_2014-2021_Liq_20230424.csv
Target rows: 4139
Year rows: 4139
WARNING: unexpected separator: 1
WARNING: empty txt: 1
rows: 4139, columns: 1

1. Impost sobre la renda
 2021
2. Sobre la renda
 2014 2015 2016 2017 2018 2019 2020
[code: 10  remaining: 12]  Which one (1-2)[1]:

1. Sobre transmissions patrimonials i actes jurídics documentats
 2014 2015 2016 2017 2018 2019 2020
2. Transmissions patrimonials i actes jurídics documentats
 2021
[code: 20  remaining: 11]  Which one (1-2)[2]: 2

1. De l'Administració General de l'Estat
 2020 2021
2. De l'Estat
 2014 2015 2016 2017 2018 2019
[code: 40  remaining: 10]  Which one (1-2)[1]:

1. D'empreses públiques i d'altres ens públics de la C.
 2020
2. Del Sector Públic Instrumental i altres ens del Sector Públic de la
+ Comunitat
 2021
3. Del sector públic instrumental i d'altres ens públics de la C.
 2014 2016 2017 2018 2019
[code: 44  remaining: 9]  Which one (1-3)[2]:

...
[download]

As a bonus, as user input accept also a kind of "class" specification. For example, "1,3:4;2:6". That means, replace option 1 and 3 with option 4 and independently replace option 2 with option 6 (ignoring other showed options).

Additionally, offer the option to skip the actual case (modifying nothing) going on with next one and also to quit the script without writing any output.

Simply adding a new PerlMonks Discussion node, to promote an older PerlMonks Discussion node, has the same problem in that it too will fall into disuse: "I want more monkquips" had various responses throughout 2009; then one in each of 2010, 2011 & 2015.

Accordingly, I'm throwing out the idea for discussion of having a Quip Quest. I envisage this would be set up similar to the poll idea quest YYYY, in the Quest category (which appears at the top of the The Monastery Gates).

A How do I create a Quip? page would have comparable information to the How do I create a Poll? page. Here's a non-exhaustive list:

• Maximun length of quip.
• Intended quip target location: XP Nodelet, Chatterbox, ...
• Optional background & foreground colours. I would stipulate that this option requires both colours and that they be contrasting (to avoid issues with Display Settings custom colours).
• Links to related threads; e.g.
  • I want more monkquips
  • Random quotes in the top left corner
  • Monk's quips

The suggested quip(s) should perhaps be within <spoiler>...</spoiler> tags — I'm not sure about this requirement.

Looking forward to your thoughts on this.