小编tra*_*loo的帖子

我有一个实体类

public class City
{
    [BsonId]
    public string id { get; set; }
    public int value { get; set; }        
}

我创建了一个 web 服务，并且有一个 webmethod 作为

[WebMethod]
    public List<City> linQuery()
    {
        MongoConn dao = new MongoConn();
        MongoServer mongo = dao.getConnection();
        List<City> list = new List<City>();

        string dbName = dao.dbName();
        var db = mongo.GetDatabase(dbName);

        Console.WriteLine(db);
        using (mongo.RequestStart(db))
        {
            var collection = db.GetCollection<City>("cityMap");
            IQueryable<City> cities = collection.AsQueryable().Where(c => c.value > 1200);
            foreach (City city in cities)
                list.Add(city);
            return list;
        }
    }

当我运行该服务时，出现该错误： …

在我的项目中，我正在将数据从MongoDB传输到SparkSQL表以进行基于SQL的查询。但是Spark SQL让我创建了临时文件。当我要查询某些内容时，执行时间非常长，因为数据传输和映射操作会花费太多时间。

那么，我可以减少执行时间吗？我可以创建永久性Spark SQL表吗？我可以使用JDBC查询永久表吗？

我要添加代码和执行时间结果。我正在独立模式下执行所有操作。

package com.mongodb.spark.sql;

import java.util.List;

import org.apache.hadoop.conf.Configuration;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import org.bson.BSONObject;

import com.mongodb.hadoop.MongoInputFormat;
import com.mongodb.spark.demo.Observation;
import com.mongodb.spark.demo.Sensor;

import scala.Tuple2;

public class SparkSqlMongo {

public static void main(String[] args) {

    Configuration conf = new Configuration();

    conf.set("mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat");
    conf.set("mongo.input.uri", "mongodb://localhost:27017/test.observations");

    Configuration sensConf = new Configuration();

    sensConf.set("mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat");
    sensConf.set("mongo.input.uri", "mongodb://localhost:27017/test.sens");

    SparkConf sconf = new SparkConf().setMaster("local[2]").setAppName("SQL DENEME").set("nsmc.connection.host",
            "mongodb:");

    JavaSparkContext sc = new JavaSparkContext(sconf);
    SQLContext sql = new SQLContext(sc); …

我是Hive和Hadoop的新手.在我的教程中,我想创建表格为

import java.sql.SQLException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;

public class HiveCreateDb {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";

public static void main(String[] args) {
    // Register driver and create driver instance

    try {
        Class.forName(driverName);

        // get connection

        Connection con = DriverManager.getConnection("jdbc:hive2://localhost:10000/mydb", "root", "");
        Statement stmt = con.createStatement();


        stmt.executeQuery("CREATE TABLE IF NOT EXISTS " + " employee ( e_id int, name String, "
                + " salary String)" + " STORED AS TEXTFILE");

        System.out.println("Operation done successfully.");

        con.close();
    } catch …

我想只打印文件名而不打印目录名称.所以,我实现了这个功能

void list_file(char* directory){
  DIR *d;
  struct dirent *dir;
  d = opendir(directory);
  if (d)
  {
    while ((dir = readdir(d)) != NULL)
    {
        printf("%c", dir->d_name[(int)strlen(dir->d_name)]);
        if(dir->d_name[(int)strlen(dir->d_name)-2] != '/')
            printf("%s\n", dir->d_name);
    }

    closedir(d);
  }
}

我检查了目录名以'/'字符结尾.所以,我检查过如果名字的末尾有'/'字符,不要打印那个名字,但是当我运行该功能时,所有这些都打印在选定的目录中？

你能告诉我如何检查目录名称的结尾？