DataTables:使用多个表,分组和html内容加速服务器端处理？

Question

DataTables:使用多个表,分组和html内容加速服务器端处理？

com*_*ral 6 php mysql performance server-side datatables

我正在使用数据表来显示来自多个mySQL表的数据(实际上是7个).目前确实没有太多数据,但我看到"显示7个条目中的1到7个(从642,660,480个条目中过滤掉)." 只显示7个条目需要20秒.一旦我真正开始向数据库添加大量内容,我确信这将无法使用.

我确信有更好的方法来完成我想要做的事情,但这是我能够让它发挥作用的唯一方法.

这是我的服务器端脚本:

$q1 = "'";
$q2 = '"';

$order_id = "CONCAT( ".$q2."<input type='hidden' id='order_id' value='".$q2.", o.id, ".$q2."'><a href='order_details.php?id=".$q2.", o.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> &nbsp;".$q2.", o.id )";
$patient_name = "CONCAT( ".$q2."<input type='hidden' id='patient_name' value='".$q2.", p.first_name, ' ', p.last_name, ".$q2."'><input type='hidden' id='patient_id' value='".$q2.", p.id, ".$q2."'><input type='hidden' id='patient_ssn' value='".$q2.", p.ssn, ".$q2."'><a href='patient_details.php?id=".$q2.", p.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> &nbsp;".$q2.", p.first_name, ' ', p.last_name )";
$doc_name = "CONCAT( ".$q2."<input type='hidden' id='doctor_name' value='".$q2.", d.first_name, ' ', d.last_name, ".$q2."'><input type='hidden' id='doctor_id' value='".$q2.", d.id, ".$q2."'><a href='doctor_details.php?id=".$q2.", d.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> &nbsp;".$q2.", d.first_name, ' ', d.last_name )";
$order_date = "FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y')";
$tests = "GROUP_CONCAT(t.name SEPARATOR ', ')";

$aColumns = array($order_id, $order_date, $doc_name, $patient_name, $tests, 'o.status');

/* Indexed column (used for fast and accurate table cardinality) */
$sIndexColumn = "o.id";

/* DB table to use */
$sTable = "`orders` o, `patients` p, `doctors` d, `tests_ordered` tst, `tests` t, `users` u, `events` e";

$sWhere = "WHERE p.id = o.patient_id AND d.id = o.doctor_id AND tst.order_id = o.id AND t.id = tst.test_id AND u.username = o.assigned_username AND e.event_id = o.event_id";
$order_status = isset($_GET['status']) ? $_GET['status'] : 'all';
if($order_status == 'all'){

}else{
    $sWhere .= " AND (o.status='Complete' OR o.status='$order_status')";
}

$sGroupBy =  "GROUP BY o.id";

/* Database connection information */
$gaSql['user']       = DB_USER;
$gaSql['password']   = DB_PASSWORD;
$gaSql['db']         = DB_NAME;
$gaSql['server']     = DB_SERVER;

/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
 * If you just want to use the basic configuration for DataTables with PHP server-side, there is
 * no need to edit below this line
 */

/* 
 * MySQL connection
 */
$gaSql['link'] =  mysql_pconnect( $gaSql['server'], $gaSql['user'], $gaSql['password']  ) or
    die( 'Could not open connection to server' );

mysql_select_db( $gaSql['db'], $gaSql['link'] ) or 
    die( 'Could not select database '. $gaSql['db'] );

/* 
 * Paging
 */
$sLimit = "";
if ( isset( $_GET['iDisplayStart'] ) && $_GET['iDisplayLength'] != '-1' )
{
    $sLimit = "LIMIT ".mysql_real_escape_string( $_GET['iDisplayStart'] ).", ".
        mysql_real_escape_string( $_GET['iDisplayLength'] );
}

/*
 * Ordering
 */
if ( isset( $_GET['iSortCol_0'] ) )
{
    $sOrder = "ORDER BY  ";
    for ( $i=0 ; $i<intval( $_GET['iSortingCols'] ) ; $i++ )
    {
        if ( $_GET[ 'bSortable_'.intval($_GET['iSortCol_'.$i]) ] == "true" )
        {
            $sOrder .= $aColumns[ intval( $_GET['iSortCol_'.$i] ) ]."
                ".mysql_real_escape_string( $_GET['sSortDir_'.$i] ) .", ";
        }
    }

    $sOrder = substr_replace( $sOrder, "", -2 );
    if ( $sOrder == "ORDER BY" )
    {
        $sOrder = "";
    }
}

/* 
 * Filtering
 * NOTE this does not match the built-in DataTables filtering which does it
 * word by word on any field. It's possible to do here, but concerned about efficiency
 * on very large tables, and MySQL's regex functionality is very limited
 */

if ( $_GET['sSearch'] != "" )
{
    $sWhere .= " AND (";
    for ( $i=0 ; $i<count($aColumns) ; $i++ )
    {
if($i!=4){ //skip tests column
        $sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string( $_GET['sSearch'] )."%' OR ";
}
    }
    $sWhere = substr_replace( $sWhere, "", -3 );
    $sWhere .= ')';
}

/* Individual column filtering */
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
if($i!=4){ //skip tests column
    if ( $_GET['bSearchable_'.$i] == "true" && $_GET['sSearch_'.$i] != '' )
    {
        if ( $sWhere == "" )
        {
            $sWhere = "WHERE ";
        }
        else
        {
            $sWhere .= " AND ";
        }
        $sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string($_GET['sSearch_'.$i])."%' ";
    }
}
}


/*
 * SQL queries
 * Get data to display
 */
$sQuery = "
    SELECT SQL_CALC_FOUND_ROWS ".str_replace(" , ", " ", implode(", ", $aColumns))."
    FROM   $sTable
    $sWhere
    $sGroupBy 
    $sOrder
    $sLimit
";

//echo $sQuery;
//die();

$rResult = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());

/* Data set length after filtering */
$sQuery = "
    SELECT FOUND_ROWS()
";
$rResultFilterTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultFilterTotal = mysql_fetch_array($rResultFilterTotal);
$iFilteredTotal = $aResultFilterTotal[0];

/* Total data set length */
$sQuery = "
    SELECT COUNT(".$sIndexColumn.")
    FROM   $sTable
";
$rResultTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultTotal = mysql_fetch_array($rResultTotal);
$iTotal = $aResultTotal[0];

//added to hide filtering   
//$iTotal = $iFilteredTotal;

/*
 * Output
 */
$output = array(
    "sEcho" => intval($_GET['sEcho']),
    "iTotalRecords" => $iTotal,
    "iTotalDisplayRecords" => $iFilteredTotal,
    "aaData" => array()
);

while ( $aRow = mysql_fetch_array( $rResult ) )
{
    $row = array();
    for ( $i=0 ; $i<count($aColumns) ; $i++ )
    {
        if ( $aColumns[$i] != ' ' )
        {
            /* General output */
            $row[] = $aRow[$i];
        }
    }
    $output['aaData'][] = $row;
}

echo json_encode( $output );

Run Code Online (Sandbox Code Playgroud)

以下是服务器端脚本生成的查询:

SELECT SQL_CALC_FOUND_ROWS 
    CONCAT( "<input type='hidden' id='order_id' value='", o.id, "'><a href='order_details.php?id=", o.id, "'><img src='search.png' border='0'></a> &nbsp;", o.id ),
    FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y'),
    CONCAT( "<input type='hidden' id='doctor_name' value='", d.first_name, ' ', d.last_name, "'><input type='hidden' id='doctor_id' value='", d.id, "'><a href='doctor_details.php?id=", d.id, "'><img src='search.png' border='0'></a> &nbsp;", d.first_name, ' ', d.last_name ),
    CONCAT( "<input type='hidden' id='patient_name' value='", p.first_name, ' ', p.last_name, "'><input type='hidden' id='patient_id' value='", p.id, "'><input type='hidden' id='patient_ssn' value='", p.ssn, "'><a href='patient_details.php?id=", p.id, "'><img src='search.png' border='0'></a> &nbsp;", p.first_name, ' ', p.last_name ), GROUP_CONCAT(t.name SEPARATOR ', '),
    o.status
FROM `orders` o, `patients` p, `doctors` d, `tests_ordered` tst, `tests` t, `users` u, `events` e
WHERE p.id = o.patient_id AND d.id = o.doctor_id AND tst.order_id = o.id AND t.id = tst.test_id AND u.username = o.assigned_username AND e.event_id = o.event_id AND (o.status='Complete' OR o.status='Draft')
GROUP BY o.id

Run Code Online (Sandbox Code Playgroud)

我正在尝试确定如何在不破坏数据表的搜索和排序功能的情况下优化它.我已经为所有表创建了索引并设置了主键,因为我知道如何.有没有办法使用JOIN？

这是EXPLAIN语句的输出:

id  select_type table   type    possible_keys   key key_len ref rows    Extra
1   SIMPLE  u   index   NULL    PRIMARY 32  NULL    5   Using index; Using temporary; Using filesort
1   SIMPLE  o   ALL PRIMARY,patient_id,doctor_id,event_id,assigned_use...   NULL    NULL    NULL    6   Using where
1   SIMPLE  d   eq_ref  PRIMARY PRIMARY 4   pasdbadmin.o.doctor_id  1    
1   SIMPLE  e   eq_ref  PRIMARY PRIMARY 4   pasdbadmin.o.event_id   1   Using index
1   SIMPLE  tst ref order_id,test_id    order_id    4   pasdbadmin.o.id 1    
1   SIMPLE  t   eq_ref  PRIMARY PRIMARY 4   pasdbadmin.tst.test_id  1    
1   SIMPLE  p   eq_ref  PRIMARY PRIMARY 4   pasdbadmin.o.patient_id 1

Run Code Online (Sandbox Code Playgroud)

更新:

问题似乎是使用包括查询中的用户和事件表(两者都没有实际使用)的问题.以下是执行速度更快的修订代码:

$q1 = "'";
$q2 = '"';

$order_id = "CONCAT( ".$q2."<input type='hidden' id='order_id' value='".$q2.", o.id, ".$q2."'><a href='order_details.php?id=".$q2.", o.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> &nbsp;".$q2.", o.id )";
$patient_name = "CONCAT( ".$q2."<input type='hidden' id='patient_name' value='".$q2.", p.first_name, ' ', p.last_name, ".$q2."'><input type='hidden' id='patient_id' value='".$q2.", p.id, ".$q2."'><input type='hidden' id='patient_ssn' value='".$q2.", p.ssn, ".$q2."'><a href='patient_details.php?id=".$q2.", p.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> &nbsp;".$q2.", p.first_name, ' ', p.last_name )";
$doc_name = "CONCAT( ".$q2."<input type='hidden' id='doctor_name' value='".$q2.", d.first_name, ' ', d.last_name, ".$q2."'><input type='hidden' id='doctor_id' value='".$q2.", d.id, ".$q2."'><a href='doctor_details.php?id=".$q2.", d.id, ".$q2."'><img src='https://pas.greysignal.com/img/search.png' border='0'></a> &nbsp;".$q2.", d.first_name, ' ', d.last_name )";
$order_date = "FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y')";
$tests = "GROUP_CONCAT(t.name SEPARATOR ', ')";

$aColumns = array($order_id, $order_date, $doc_name, $patient_name, $tests, 'o.status');

/* Indexed column (used for fast and accurate table cardinality) */
$sIndexColumn = "o.id";

/* DB table to use */
$sTable = "`orders` o, `patients` p, `doctors` d, `tests_ordered` tst, `tests` t";

$sWhere = "WHERE p.id = o.patient_id AND d.id = o.doctor_id AND tst.order_id = o.id AND t.id = tst.test_id";
$order_status = isset($_GET['status']) ? $_GET['status'] : 'all';
if($order_status == 'all'){

}else{
    $sWhere .= " AND (o.status='Complete' OR o.status='$order_status')";
}

$sJoin = "";

$sGroupBy =  "GROUP BY o.id";

/* Database connection information */
$gaSql['user']       = DB_USER;
$gaSql['password']   = DB_PASSWORD;
$gaSql['db']         = DB_NAME;
$gaSql['server']     = DB_SERVER;

/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
 * If you just want to use the basic configuration for DataTables with PHP server-side, there is
 * no need to edit below this line
 */

/* 
 * MySQL connection
 */
$gaSql['link'] =  mysql_pconnect( $gaSql['server'], $gaSql['user'], $gaSql['password']  ) or
    die( 'Could not open connection to server' );

mysql_select_db( $gaSql['db'], $gaSql['link'] ) or 
    die( 'Could not select database '. $gaSql['db'] );

/* 
 * Paging
 */
$sLimit = "";
if ( isset( $_GET['iDisplayStart'] ) && $_GET['iDisplayLength'] != '-1' )
{
    $sLimit = "LIMIT ".mysql_real_escape_string( $_GET['iDisplayStart'] ).", ".
        mysql_real_escape_string( $_GET['iDisplayLength'] );
}

/*
 * Ordering
 */
if ( isset( $_GET['iSortCol_0'] ) )
{
    $sOrder = "ORDER BY  ";
    for ( $i=0 ; $i<intval( $_GET['iSortingCols'] ) ; $i++ )
    {
        if ( $_GET[ 'bSortable_'.intval($_GET['iSortCol_'.$i]) ] == "true" )
        {
            $sOrder .= $aColumns[ intval( $_GET['iSortCol_'.$i] ) ]."
                ".mysql_real_escape_string( $_GET['sSortDir_'.$i] ) .", ";
        }
    }

    $sOrder = substr_replace( $sOrder, "", -2 );
    if ( $sOrder == "ORDER BY" )
    {
        $sOrder = "";
    }
}

/* 
 * Filtering
 * NOTE this does not match the built-in DataTables filtering which does it
 * word by word on any field. It's possible to do here, but concerned about efficiency
 * on very large tables, and MySQL's regex functionality is very limited
 */

if ( $_GET['sSearch'] != "" )
{
    $sWhere .= " AND (";
    for ( $i=0 ; $i<count($aColumns) ; $i++ )
    {
        if($i!=4){ //skip tests column
            $sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string( $_GET['sSearch'] )."%' OR ";
        }
    }
    $sWhere = substr_replace( $sWhere, "", -3 );
    $sWhere .= ')';
}

/* Individual column filtering */
for ( $i=0 ; $i<count($aColumns) ; $i++ )
{
    if($i!=4){ //skip tests column
        if ( $_GET['bSearchable_'.$i] == "true" && $_GET['sSearch_'.$i] != '' )
        {
            if ( $sWhere == "" )
            {
                $sWhere = "WHERE ";
            }
            else
            {
                $sWhere .= " AND ";
            }
            $sWhere .= $aColumns[$i]." LIKE '%".mysql_real_escape_string($_GET['sSearch_'.$i])."%' ";
        }
    }
}


/*
 * SQL queries
 * Get data to display
 */
$sQuery = "
    SELECT SQL_CALC_FOUND_ROWS ".str_replace(" , ", " ", implode(", ", $aColumns))."
    FROM   $sTable
    $sWhere
    $sJoin
    $sGroupBy 
    $sOrder
    $sLimit
";

$filename = __DIR__.DIRECTORY_SEPARATOR."sql_log.txt";
file_put_contents($filename, $sQuery, FILE_APPEND);

$rResult = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());

/* Data set length after filtering */
$sQuery = "
    SELECT FOUND_ROWS()
";
$rResultFilterTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultFilterTotal = mysql_fetch_array($rResultFilterTotal);
$iFilteredTotal = $aResultFilterTotal[0];

/* Total data set length */
$sQuery = "
    SELECT COUNT(".$sIndexColumn.")
    FROM   $sTable
";
$rResultTotal = mysql_query( $sQuery, $gaSql['link'] ) or die(mysql_error());
$aResultTotal = mysql_fetch_array($rResultTotal);
$iTotal = $aResultTotal[0];

//added to hide filtering   
//$iTotal = $iFilteredTotal;

/*
 * Output
 */
$output = array(
    "sEcho" => intval($_GET['sEcho']),
    "iTotalRecords" => $iTotal,
    "iTotalDisplayRecords" => $iFilteredTotal,
    "aaData" => array()
);

while ( $aRow = mysql_fetch_array( $rResult ) )
{
    $row = array();
    for ( $i=0 ; $i<count($aColumns) ; $i++ )
    {
        if ( $aColumns[$i] != ' ' )
        {
            /* General output */
            $row[] = $aRow[$i];
        }
    }
    $output['aaData'][] = $row;
}

echo json_encode( $output );

Run Code Online (Sandbox Code Playgroud)

如果我要使用JOINS,更新的查询将是:

 SELECT SQL_CALC_FOUND_ROWS 
     CONCAT( "<input type='hidden' id='order_id' value='", o.id, "'><a href='order_details.php?id=", o.id, "'><img src='search.png' border='0'></a> &nbsp;", o.id ),
     FROM_UNIXTIME(o.created_timestamp, '%m/%e/%Y'),
     CONCAT( "<input type='hidden' id='doctor_name' value='", d.first_name, ' ', d.last_name, "'><input type='hidden' id='doctor_id' value='", d.id, "'><a href='doctor_details.php?id=", d.id, "'><img src='search.png' border='0'></a> &nbsp;", d.first_name, ' ', d.last_name ),
     CONCAT( "<input type='hidden' id='patient_name' value='", p.first_name, ' ', p.last_name, "'><input type='hidden' id='patient_id' value='", p.id, "'><input type='hidden' id='patient_ssn' value='", p.ssn, "'><a href='patient_details.php?id=", p.id, "'><img src='search.png' border='0'></a> &nbsp;", p.first_name, ' ', p.last_name ),
     GROUP_CONCAT(t.name SEPARATOR ', '),
     o.status
 FROM `orders` o
 JOIN `doctors` d ON d.id = o.doctor_id
 JOIN `patients` p ON p.id = o.patient_id
 JOIN `tests_ordered` tst ON tst.order_id = o.id
 JOIN `tests` t ON t.id = tst.test_id 
 WHERE o.status='Complete' OR o.status='Draft'
 GROUP BY o.id

Run Code Online (Sandbox Code Playgroud)

这个问题是,由于列数组等原因,使用排序和过滤时,DataTables的设计并不适合JOINS正常运行.我很乐意看到一个可以解决这个问题的解决方案.

Answer 1

Haz*_*zit 5

首先,如果要优化SQL语句,首先要去除那里的HTML垃圾.如果不出意外,它会混淆Statement的实际结构.如果你必须,你可以在优化结束时把它放回去,虽然我会认真投票反对它:你有PHP做标记.为了清楚起见,我已经养成了使用JOIN条款的习惯,我相应地改写了整个Thing.

这个过程给了我的是:

SELECT SQL_CALC_FOUND_ROWS,
    o.id, o.created_timestamp, o.status,
    d.id, d.first_name, d.last_name, 
    p.id, p.first_name, p.last_name, p.ssn 
    GROUP_CONCAT(t.name SEPARATOR ', '),
FROM `orders` o
JOIN `doctors` d ON d.id = o.doctor_id
JOIN `patients` p ON p.id = o.patient_id
JOIN `users` u ON u.username = o.assigned_username
JOIN `events` e ON e.event_id = o.event_id
JOIN `tests_ordered` tst ON tst.order_id = o.id
JOIN `tests` t ON t.id = tst.test_id 
WHERE o.status='Complete' OR o.status='Draft'
GROUP BY o.id

Run Code Online (Sandbox Code Playgroud)

这里有一些事情需要注意:

1)你的主桌是orders.这也是你正在使用AND的WHERE子句的那个你是GROUPing BY的那个.有了id主要索引和第二个索引status,这应该不会太糟糕.

2)你在其他四个表中链接我假设是外键.这些表中的两个从未实际使用:有(最有可能)没有必要加入上users和events在所有.你应该摆脱那些,顺便说一下,它也删除了文本列(用户名)上不那么出色的连接.确保剩余的表doctors并将patients其主键放在各自的id列上.

3)你有一个更复杂的加入两个表tests_ordered和tests.所有这一切都是为您提供一个串联的名称字符串,但它确实增加了GROUP BY子句的复杂性.有两种方法可以从这里开始:尝试优化这些连接或从完全选择中删除它们.

3)解决方案A要优化这些连接,请确保您已启用索引tests_ordered.order_id并打开tests_ordered.test_id主索引tests.id.您的声明应如下所示:

SELECT SQL_CALC_FOUND_ROWS,
    o.id, o.created_timestamp, o.status,
    d.id, d.first_name, d.last_name, 
    p.id, p.first_name, p.last_name, p.ssn 
    GROUP_CONCAT(t.name SEPARATOR ', '),
FROM `orders` o
JOIN `doctors` d ON d.id = o.doctor_id
JOIN `patients` p ON p.id = o.patient_id
JOIN `tests_ordered` tst ON tst.order_id = o.id
JOIN `tests` t ON t.id = tst.test_id 
WHERE o.status='Complete' OR o.status='Draft'
GROUP BY o.id

Run Code Online (Sandbox Code Playgroud)

3)解决方案B删除整个测试/ tests_ordered的东西,并将其放在单独的选择中.您的主要选择现在将如下所示:

SELECT SQL_CALC_FOUND_ROWS,
    o.id, o.created_timestamp, o.status,
    d.id, d.first_name, d.last_name, 
    p.id, p.first_name, p.last_name, p.ssn 
FROM `orders` o
JOIN `doctors` d ON d.id = o.doctor_id
JOIN `patients` p ON p.id = o.patient_id
WHERE o.status='Complete' OR o.status='Draft'

Run Code Online (Sandbox Code Playgroud)

但是,您必须每行运行一个额外的SELECT才能获得连接的t.name,或者只对当前页面上的所有订单ID执行一个SELECT.后者看起来像这样:

SELECT o.id, GROUP_CONCAT(t.name SEPARATOR ', '),
FROM `orders` o
JOIN `tests_ordered` tst ON tst.order_id = o.id
JOIN `tests` t ON t.id = tst.test_id 
WHERE o.in IN ( <put the 10 ids on your current page here, separated by commas> )
GROUP BY o.id

Run Code Online (Sandbox Code Playgroud)

解决方案A应该在一台不错的机器上运行得非常快解决方案B应该在任何机器上运行得非常快.索引外键上的直接JOINS很便宜.

4)以上选择中的任何一个都不应该在其中包含少于6.42亿个订单的数据库中返回6.42亿行,我假设您没有.因为MySQL告诉你它可能需要构建一个完整的笛卡尔积,这也解释了经验丰富的速度.这意味着你的一个直接外键加入变得混乱.最有可能的问题是加入users- 首先是无用的连接,但无论如何:检查其username列的唯一性.

归档时间：	12 年，9 月前
查看次数：	6022 次
最近记录：	12 年，9 月前